MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917623315300352 |
|---|---|
| author | Qin, Yiran Zhou, Enshen Liu, Qichang Yin, Zhenfei Sheng, Lu Zhang, Ruimao Qiao, Yu Shao, Jing |
| author_facet | Qin, Yiran Zhou, Enshen Liu, Qichang Yin, Zhenfei Sheng, Lu Zhang, Ruimao Qiao, Yu Shao, Jing |
| contents | It is a long-lasting goal to design an embodied system that can solve long-horizon open-world tasks in human-like ways. However, existing approaches usually struggle with compound difficulties caused by the logic-aware decomposition and context-aware execution of these tasks. To this end, we introduce MP5, an open-ended multimodal embodied system built upon the challenging Minecraft simulator, which can decompose feasible sub-objectives, design sophisticated situation-aware plans, and perform embodied action control, with frequent communication with a goal-conditioned active perception scheme. Specifically, MP5 is developed on top of recent advances in Multimodal Large Language Models (MLLMs), and the system is modulated into functional modules that can be scheduled and collaborated to ultimately solve pre-defined context- and process-dependent tasks. Extensive experiments prove that MP5 can achieve a 22% success rate on difficult process-dependent tasks and a 91% success rate on tasks that heavily depend on the context. Moreover, MP5 exhibits a remarkable ability to address many open-ended tasks that are entirely novel. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_07472 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception Qin, Yiran Zhou, Enshen Liu, Qichang Yin, Zhenfei Sheng, Lu Zhang, Ruimao Qiao, Yu Shao, Jing Computer Vision and Pattern Recognition It is a long-lasting goal to design an embodied system that can solve long-horizon open-world tasks in human-like ways. However, existing approaches usually struggle with compound difficulties caused by the logic-aware decomposition and context-aware execution of these tasks. To this end, we introduce MP5, an open-ended multimodal embodied system built upon the challenging Minecraft simulator, which can decompose feasible sub-objectives, design sophisticated situation-aware plans, and perform embodied action control, with frequent communication with a goal-conditioned active perception scheme. Specifically, MP5 is developed on top of recent advances in Multimodal Large Language Models (MLLMs), and the system is modulated into functional modules that can be scheduled and collaborated to ultimately solve pre-defined context- and process-dependent tasks. Extensive experiments prove that MP5 can achieve a 22% success rate on difficult process-dependent tasks and a 91% success rate on tasks that heavily depend on the context. Moreover, MP5 exhibits a remarkable ability to address many open-ended tasks that are entirely novel. |
| title | MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.07472 |