Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2510.08759 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914583941218304 |
|---|---|
| author | Qi, Yu Zhao, Haibo Guo, Ziyu Ma, Siyuan Chen, Ziyan Han, Yaokun Zhang, Renrui Lin, Zitiantao Zhu, Yizhe Xin, Shiji Huang, Yijian Hu, Boce Cheng, Kai Wang, Peiheng Liu, Jiazheng Zhang, Jiayi Zhu, Yizhe Wang, Wenqing Qin, Yiran Huang, Haojie Wong, Lawson L. S. |
| author_facet | Qi, Yu Zhao, Haibo Guo, Ziyu Ma, Siyuan Chen, Ziyan Han, Yaokun Zhang, Renrui Lin, Zitiantao Zhu, Yizhe Xin, Shiji Huang, Yijian Hu, Boce Cheng, Kai Wang, Peiheng Liu, Jiazheng Zhang, Jiayi Zhu, Yizhe Wang, Wenqing Qin, Yiran Huang, Haojie Wong, Lawson L. S. |
| contents | Understanding the capability bottlenecks of embodied multimodal large language models (MLLMs) is crucial for improving embodied agents. However, existing embodied benchmarks mainly focus on task-level evaluation and fail to provide actionable insights into the underlying causes of model failures. To address this limitation, we introduce BEAR, a benchmark that decomposes embodied tasks into 14 atomic skills for fine-grained skill-level evaluation. BEAR comprises 4,469 interleaved image-video-text samples spanning 14 skills across 6 categories, ranging from low-level perception to high-level planning. We evaluate 20 MLLMs on BEAR under a hierarchical skill-level diagnosis framework and uncover two key findings: (1) perceptual capabilities are major bottlenecks behind reasoning failures, and (2) current models suffer from unstable spatiotemporal modeling that remains largely unexposed in prior benchmarks. Motivated by these findings, we further propose BEAR-Agent, a multimodal conversational agent that augments MLLMs with visual and spatial reasoning tools. BEAR-Agent substantially improves performance across embodied skills, achieving a relative improvement of 17.5% on GPT-5 over the base model on BEAR, while also outperforming strong baselines in both simulation and real-world robotic experiments. Project page: https://bear-official66.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08759 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis Qi, Yu Zhao, Haibo Guo, Ziyu Ma, Siyuan Chen, Ziyan Han, Yaokun Zhang, Renrui Lin, Zitiantao Zhu, Yizhe Xin, Shiji Huang, Yijian Hu, Boce Cheng, Kai Wang, Peiheng Liu, Jiazheng Zhang, Jiayi Zhu, Yizhe Wang, Wenqing Qin, Yiran Huang, Haojie Wong, Lawson L. S. Computer Vision and Pattern Recognition Robotics Understanding the capability bottlenecks of embodied multimodal large language models (MLLMs) is crucial for improving embodied agents. However, existing embodied benchmarks mainly focus on task-level evaluation and fail to provide actionable insights into the underlying causes of model failures. To address this limitation, we introduce BEAR, a benchmark that decomposes embodied tasks into 14 atomic skills for fine-grained skill-level evaluation. BEAR comprises 4,469 interleaved image-video-text samples spanning 14 skills across 6 categories, ranging from low-level perception to high-level planning. We evaluate 20 MLLMs on BEAR under a hierarchical skill-level diagnosis framework and uncover two key findings: (1) perceptual capabilities are major bottlenecks behind reasoning failures, and (2) current models suffer from unstable spatiotemporal modeling that remains largely unexposed in prior benchmarks. Motivated by these findings, we further propose BEAR-Agent, a multimodal conversational agent that augments MLLMs with visual and spatial reasoning tools. BEAR-Agent substantially improves performance across embodied skills, achieving a relative improvement of 17.5% on GPT-5 over the base model on BEAR, while also outperforming strong baselines in both simulation and real-world robotic experiments. Project page: https://bear-official66.github.io/ |
| title | Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2510.08759 |