PROC2PDDL: Open-Domain Planning Representations from Texts
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916308420919296 |
|---|---|
| author | Zhang, Tianyi Zhang, Li Hou, Zhaoyi Wang, Ziyu Gu, Yuling Clark, Peter Callison-Burch, Chris Tandon, Niket |
| author_facet | Zhang, Tianyi Zhang, Li Hou, Zhaoyi Wang, Ziyu Gu, Yuling Clark, Peter Callison-Burch, Chris Tandon, Niket |
| contents | Planning in a text-based environment continues to be a major challenge for AI systems. Recent approaches have used language models to predict a planning domain definition (e.g., PDDL) but have only been evaluated in closed-domain simulated environments. To address this, we present Proc2PDDL , the first dataset containing open-domain procedural texts paired with expert-annotated PDDL representations. Using this dataset, we evaluate state-of-the-art models on defining the preconditions and effects of actions. We show that Proc2PDDL is highly challenging, with GPT-3.5's success rate close to 0% and GPT-4's around 35%. Our analysis shows both syntactic and semantic errors, indicating LMs' deficiency in both generating domain-specific prgorams and reasoning about events. We hope this analysis and dataset helps future progress towards integrating the best of LMs and formal planning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00092 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PROC2PDDL: Open-Domain Planning Representations from Texts Zhang, Tianyi Zhang, Li Hou, Zhaoyi Wang, Ziyu Gu, Yuling Clark, Peter Callison-Burch, Chris Tandon, Niket Computation and Language Planning in a text-based environment continues to be a major challenge for AI systems. Recent approaches have used language models to predict a planning domain definition (e.g., PDDL) but have only been evaluated in closed-domain simulated environments. To address this, we present Proc2PDDL , the first dataset containing open-domain procedural texts paired with expert-annotated PDDL representations. Using this dataset, we evaluate state-of-the-art models on defining the preconditions and effects of actions. We show that Proc2PDDL is highly challenging, with GPT-3.5's success rate close to 0% and GPT-4's around 35%. Our analysis shows both syntactic and semantic errors, indicating LMs' deficiency in both generating domain-specific prgorams and reasoning about events. We hope this analysis and dataset helps future progress towards integrating the best of LMs and formal planning. |
| title | PROC2PDDL: Open-Domain Planning Representations from Texts |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.00092 |