Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.01031 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916772070817792 |
|---|---|
| author | Qiao, Yanyuan Hong, Haodong Lyu, Wenqi An, Dong Zhang, Siqi Xie, Yutong Wang, Xinyu Wu, Qi |
| author_facet | Qiao, Yanyuan Hong, Haodong Lyu, Wenqi An, Dong Zhang, Siqi Xie, Yutong Wang, Xinyu Wu, Qi |
| contents | Multimodal Large Language Models (MLLMs) have demonstrated strong generalization in vision-language tasks, yet their ability to understand and act within embodied environments remains underexplored. We present NavBench, a benchmark to evaluate the embodied navigation capabilities of MLLMs under zero-shot settings. NavBench consists of two components: (1) navigation comprehension, assessed through three cognitively grounded tasks including global instruction alignment, temporal progress estimation, and local observation-action reasoning, covering 3,200 question-answer pairs; and (2) step-by-step execution in 432 episodes across 72 indoor scenes, stratified by spatial, cognitive, and execution complexity. To support real-world deployment, we introduce a pipeline that converts MLLMs' outputs into robotic actions. We evaluate both proprietary and open-source models, finding that GPT-4o performs well across tasks, while lighter open-source models succeed in simpler cases. Results also show that models with higher comprehension scores tend to achieve better execution performance. Providing map-based context improves decision accuracy, especially in medium-difficulty scenarios. However, most models struggle with temporal understanding, particularly in estimating progress during navigation, which may pose a key challenge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01031 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NavBench: Probing Multimodal Large Language Models for Embodied Navigation Qiao, Yanyuan Hong, Haodong Lyu, Wenqi An, Dong Zhang, Siqi Xie, Yutong Wang, Xinyu Wu, Qi Computer Vision and Pattern Recognition Multimodal Large Language Models (MLLMs) have demonstrated strong generalization in vision-language tasks, yet their ability to understand and act within embodied environments remains underexplored. We present NavBench, a benchmark to evaluate the embodied navigation capabilities of MLLMs under zero-shot settings. NavBench consists of two components: (1) navigation comprehension, assessed through three cognitively grounded tasks including global instruction alignment, temporal progress estimation, and local observation-action reasoning, covering 3,200 question-answer pairs; and (2) step-by-step execution in 432 episodes across 72 indoor scenes, stratified by spatial, cognitive, and execution complexity. To support real-world deployment, we introduce a pipeline that converts MLLMs' outputs into robotic actions. We evaluate both proprietary and open-source models, finding that GPT-4o performs well across tasks, while lighter open-source models succeed in simpler cases. Results also show that models with higher comprehension scores tend to achieve better execution performance. Providing map-based context improves decision accuracy, especially in medium-difficulty scenarios. However, most models struggle with temporal understanding, particularly in estimating progress during navigation, which may pose a key challenge. |
| title | NavBench: Probing Multimodal Large Language Models for Embodied Navigation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.01031 |