Using Left and Right Brains Together: Towards Vision and Language Planning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911778312552448 |
|---|---|
| author | Cen, Jun Wu, Chenfei Liu, Xiao Yin, Shengming Pei, Yixuan Yang, Jinglong Chen, Qifeng Duan, Nan Zhang, Jianguo |
| author_facet | Cen, Jun Wu, Chenfei Liu, Xiao Yin, Shengming Pei, Yixuan Yang, Jinglong Chen, Qifeng Duan, Nan Zhang, Jianguo |
| contents | Large Language Models (LLMs) and Large Multi-modality Models (LMMs) have demonstrated remarkable decision masking capabilities on a variety of tasks. However, they inherently operate planning within the language space, lacking the vision and spatial imagination ability. In contrast, humans utilize both left and right hemispheres of the brain for language and visual planning during the thinking process. Therefore, we introduce a novel vision-language planning framework in this work to perform concurrent visual and language planning for tasks with inputs of any form. Our framework incorporates visual planning to capture intricate environmental details, while language planning enhances the logical coherence of the overall system. We evaluate the effectiveness of our framework across vision-language tasks, vision-only tasks, and language-only tasks. The results demonstrate the superior performance of our approach, indicating that the integration of visual and language planning yields better contextually aware task execution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_10534 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Using Left and Right Brains Together: Towards Vision and Language Planning Cen, Jun Wu, Chenfei Liu, Xiao Yin, Shengming Pei, Yixuan Yang, Jinglong Chen, Qifeng Duan, Nan Zhang, Jianguo Computer Vision and Pattern Recognition Large Language Models (LLMs) and Large Multi-modality Models (LMMs) have demonstrated remarkable decision masking capabilities on a variety of tasks. However, they inherently operate planning within the language space, lacking the vision and spatial imagination ability. In contrast, humans utilize both left and right hemispheres of the brain for language and visual planning during the thinking process. Therefore, we introduce a novel vision-language planning framework in this work to perform concurrent visual and language planning for tasks with inputs of any form. Our framework incorporates visual planning to capture intricate environmental details, while language planning enhances the logical coherence of the overall system. We evaluate the effectiveness of our framework across vision-language tasks, vision-only tasks, and language-only tasks. The results demonstrate the superior performance of our approach, indicating that the integration of visual and language planning yields better contextually aware task execution. |
| title | Using Left and Right Brains Together: Towards Vision and Language Planning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.10534 |