Vision-Language Interpreter for Robot Task Planning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916131713843200 |
|---|---|
| author | Shirai, Keisuke Beltran-Hernandez, Cristian C. Hamaya, Masashi Hashimoto, Atsushi Tanaka, Shohei Kawaharazuka, Kento Tanaka, Kazutoshi Ushiku, Yoshitaka Mori, Shinsuke |
| author_facet | Shirai, Keisuke Beltran-Hernandez, Cristian C. Hamaya, Masashi Hashimoto, Atsushi Tanaka, Shohei Kawaharazuka, Kento Tanaka, Kazutoshi Ushiku, Yoshitaka Mori, Shinsuke |
| contents | Large language models (LLMs) are accelerating the development of language-guided robot planners. Meanwhile, symbolic planners offer the advantage of interpretability. This paper proposes a new task that bridges these two trends, namely, multimodal planning problem specification. The aim is to generate a problem description (PD), a machine-readable file used by the planners to find a plan. By generating PDs from language instruction and scene observation, we can drive symbolic planners in a language-guided framework. We propose a Vision-Language Interpreter (ViLaIn), a new framework that generates PDs using state-of-the-art LLM and vision-language models. ViLaIn can refine generated PDs via error message feedback from the symbolic planner. Our aim is to answer the question: How accurately can ViLaIn and the symbolic planner generate valid robot plans? To evaluate ViLaIn, we introduce a novel dataset called the problem description generation (ProDG) dataset. The framework is evaluated with four new evaluation metrics. Experimental results show that ViLaIn can generate syntactically correct problems with more than 99\% accuracy and valid plans with more than 58\% accuracy. Our code and dataset are available at https://github.com/omron-sinicx/ViLaIn. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_00967 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Vision-Language Interpreter for Robot Task Planning Shirai, Keisuke Beltran-Hernandez, Cristian C. Hamaya, Masashi Hashimoto, Atsushi Tanaka, Shohei Kawaharazuka, Kento Tanaka, Kazutoshi Ushiku, Yoshitaka Mori, Shinsuke Robotics Artificial Intelligence Computation and Language Large language models (LLMs) are accelerating the development of language-guided robot planners. Meanwhile, symbolic planners offer the advantage of interpretability. This paper proposes a new task that bridges these two trends, namely, multimodal planning problem specification. The aim is to generate a problem description (PD), a machine-readable file used by the planners to find a plan. By generating PDs from language instruction and scene observation, we can drive symbolic planners in a language-guided framework. We propose a Vision-Language Interpreter (ViLaIn), a new framework that generates PDs using state-of-the-art LLM and vision-language models. ViLaIn can refine generated PDs via error message feedback from the symbolic planner. Our aim is to answer the question: How accurately can ViLaIn and the symbolic planner generate valid robot plans? To evaluate ViLaIn, we introduce a novel dataset called the problem description generation (ProDG) dataset. The framework is evaluated with four new evaluation metrics. Experimental results show that ViLaIn can generate syntactically correct problems with more than 99\% accuracy and valid plans with more than 58\% accuracy. Our code and dataset are available at https://github.com/omron-sinicx/ViLaIn. |
| title | Vision-Language Interpreter for Robot Task Planning |
| topic | Robotics Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2311.00967 |