Evaluating Language Models Planning Capabilities on Goal Ordering Challenges

Planning involves the composition of primitive actions to achieve specific goals within a given environment. Classical planning research has well-established different types of goal-ordering challenges which have implications on the planning heuristics. In this study, we investigate the performance of Large Language Models (LLMs) in identifying if an order between two goals hold. We distinguish between three types of goal orderings challenges: reasonable, necessary, and optimal. Our findings reveal that LLMs predominantly struggle with reasonable goal ordering tasks compared to necessary and optimal goal orderings. Advancing this area could lead to improvements in the planning abilities of LLMs.