MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910545207099392 |
|---|---|
| author | Liu, Yang Song, Xinshuai Jiang, Kaixuan Chen, Weixing Luo, Jingzhou Li, Guanbin Lin, Liang |
| author_facet | Liu, Yang Song, Xinshuai Jiang, Kaixuan Chen, Weixing Luo, Jingzhou Li, Guanbin Lin, Liang |
| contents | With the surge in the development of large language models, embodied intelligence has attracted increasing attention. Nevertheless, prior works on embodied intelligence typically encode scene or historical memory in an unimodal manner, either visual or linguistic, which complicates the alignment of the model's action planning with embodied control. To overcome this limitation, we introduce the Multimodal Embodied Interactive Agent (MEIA), capable of translating high-level tasks expressed in natural language into a sequence of executable actions. Specifically, we propose a novel Multimodal Environment Memory (MEM) module, facilitating the integration of embodied control with large models through the visual-language memory of scenes. This capability enables MEIA to generate executable action plans based on diverse requirements and the robot's capabilities. Furthermore, we construct an embodied question answering dataset based on a dynamic virtual cafe environment with the help of the large language model. In this virtual environment, we conduct several experiments, utilizing multiple large models through zero-shot learning, and carefully design scenarios for various situations. The experimental results showcase the promising performance of our MEIA in various embodied interactive tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_00290 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments Liu, Yang Song, Xinshuai Jiang, Kaixuan Chen, Weixing Luo, Jingzhou Li, Guanbin Lin, Liang Computer Vision and Pattern Recognition With the surge in the development of large language models, embodied intelligence has attracted increasing attention. Nevertheless, prior works on embodied intelligence typically encode scene or historical memory in an unimodal manner, either visual or linguistic, which complicates the alignment of the model's action planning with embodied control. To overcome this limitation, we introduce the Multimodal Embodied Interactive Agent (MEIA), capable of translating high-level tasks expressed in natural language into a sequence of executable actions. Specifically, we propose a novel Multimodal Environment Memory (MEM) module, facilitating the integration of embodied control with large models through the visual-language memory of scenes. This capability enables MEIA to generate executable action plans based on diverse requirements and the robot's capabilities. Furthermore, we construct an embodied question answering dataset based on a dynamic virtual cafe environment with the help of the large language model. In this virtual environment, we conduct several experiments, utilizing multiple large models through zero-shot learning, and carefully design scenarios for various situations. The experimental results showcase the promising performance of our MEIA in various embodied interactive tasks. |
| title | MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.00290 |