DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909231232319488 |
|---|---|
| author | Zhang, Xiaohan Altaweel, Zainab Hayamizu, Yohei Ding, Yan Amiri, Saeid Yang, Hao Kaminski, Andy Esselink, Chad Zhang, Shiqi |
| author_facet | Zhang, Xiaohan Altaweel, Zainab Hayamizu, Yohei Ding, Yan Amiri, Saeid Yang, Hao Kaminski, Andy Esselink, Chad Zhang, Shiqi |
| contents | Vision-language models (VLMs) have been applied to robot task planning problems, where the robot receives a task in natural language and generates plans based on visual inputs. While current VLMs have demonstrated strong vision-language understanding capabilities, their performance is still far from being satisfactory in planning tasks. At the same time, although classical task planners, such as PDDL-based, are strong in planning for long-horizon tasks, they do not work well in open worlds where unforeseen situations are common. In this paper, we propose a novel task planning and execution framework, called DKPROMPT, which automates VLM prompting using domain knowledge in PDDL for classical planning in open worlds. Results from quantitative experiments show that DKPROMPT outperforms classical planning, pure VLM-based and a few other competitive baselines in task completion rate. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_17659 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning Zhang, Xiaohan Altaweel, Zainab Hayamizu, Yohei Ding, Yan Amiri, Saeid Yang, Hao Kaminski, Andy Esselink, Chad Zhang, Shiqi Artificial Intelligence Robotics Vision-language models (VLMs) have been applied to robot task planning problems, where the robot receives a task in natural language and generates plans based on visual inputs. While current VLMs have demonstrated strong vision-language understanding capabilities, their performance is still far from being satisfactory in planning tasks. At the same time, although classical task planners, such as PDDL-based, are strong in planning for long-horizon tasks, they do not work well in open worlds where unforeseen situations are common. In this paper, we propose a novel task planning and execution framework, called DKPROMPT, which automates VLM prompting using domain knowledge in PDDL for classical planning in open worlds. Results from quantitative experiments show that DKPROMPT outperforms classical planning, pure VLM-based and a few other competitive baselines in task completion rate. |
| title | DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning |
| topic | Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2406.17659 |