NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916278571106304 |
|---|---|
| author | Zheng, Huaixiu Steven Mishra, Swaroop Zhang, Hugh Chen, Xinyun Chen, Minmin Nova, Azade Hou, Le Cheng, Heng-Tze Le, Quoc V. Chi, Ed H. Zhou, Denny |
| author_facet | Zheng, Huaixiu Steven Mishra, Swaroop Zhang, Hugh Chen, Xinyun Chen, Minmin Nova, Azade Hou, Le Cheng, Heng-Tze Le, Quoc V. Chi, Ed H. Zhou, Denny |
| contents | We introduce NATURAL PLAN, a realistic planning benchmark in natural language containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar Scheduling. We focus our evaluation on the planning capabilities of LLMs with full information on the task, by providing outputs from tools such as Google Flights, Google Maps, and Google Calendar as contexts to the models. This eliminates the need for a tool-use environment for evaluating LLMs on Planning. We observe that NATURAL PLAN is a challenging benchmark for state of the art models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only achieve 31.1% and 34.8% solve rate respectively. We find that model performance drops drastically as the complexity of the problem increases: all models perform below 5% when there are 10 cities, highlighting a significant gap in planning in natural language for SoTA LLMs. We also conduct extensive ablation studies on NATURAL PLAN to further shed light on the (in)effectiveness of approaches such as self-correction, few-shot generalization, and in-context planning with long-contexts on improving LLM planning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04520 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NATURAL PLAN: Benchmarking LLMs on Natural Language Planning Zheng, Huaixiu Steven Mishra, Swaroop Zhang, Hugh Chen, Xinyun Chen, Minmin Nova, Azade Hou, Le Cheng, Heng-Tze Le, Quoc V. Chi, Ed H. Zhou, Denny Computation and Language Artificial Intelligence We introduce NATURAL PLAN, a realistic planning benchmark in natural language containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar Scheduling. We focus our evaluation on the planning capabilities of LLMs with full information on the task, by providing outputs from tools such as Google Flights, Google Maps, and Google Calendar as contexts to the models. This eliminates the need for a tool-use environment for evaluating LLMs on Planning. We observe that NATURAL PLAN is a challenging benchmark for state of the art models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only achieve 31.1% and 34.8% solve rate respectively. We find that model performance drops drastically as the complexity of the problem increases: all models perform below 5% when there are 10 cities, highlighting a significant gap in planning in natural language for SoTA LLMs. We also conduct extensive ablation studies on NATURAL PLAN to further shed light on the (in)effectiveness of approaches such as self-correction, few-shot generalization, and in-context planning with long-contexts on improving LLM planning. |
| title | NATURAL PLAN: Benchmarking LLMs on Natural Language Planning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.04520 |