MindCube: Spatial Mental Modeling from Limited Views
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911554679603200 |
|---|---|
| author | Wang, Qineng Yin, Baiqiao Zhang, Pingyue Zhang, Jianshu Wang, Kangrui Wang, Zihan Zhang, Jieyu Chandrasegaran, Keshigeyan Liu, Han Krishna, Ranjay Xie, Saining Wu, Jiajun Fei-Fei, Li Li, Manling |
| author_facet | Wang, Qineng Yin, Baiqiao Zhang, Pingyue Zhang, Jianshu Wang, Kangrui Wang, Zihan Zhang, Jieyu Chandrasegaran, Keshigeyan Liu, Han Krishna, Ranjay Xie, Saining Wu, Jiajun Fei-Fei, Li Li, Manling |
| contents | Can Vision-Language Models (VLMs) imagine the full scene from just a few views, like humans do? Humans form spatial mental models naturally, internal representations of unseen space, to reason about layout, perspective, and motion. Our MindCube benchmark with 21,154 questions across 3,268 images exposes this critical gap, where existing VLMs exhibit near-random performance. Using MindCube, we systematically evaluate how well VLMs build robust spatial mental models through representing positions (cognitive mapping), orientations (perspective-taking), and dynamics (mental simulation for "what-if" movements). We then explore three approaches to help approximate spatial mental models in VLMs, focusing on incorporating unseen intermediate views, natural language reasoning chains, and cognitive maps. The significant improvement comes from a synergistic approach, "map-then-reason", that jointly trains the model to first generate a cognitive map and then reason upon it. By training models to reason over these internal maps, we boosted accuracy from 37.8% to 57.8% (+20.0%). Adding reinforcement learning pushed performance even further to 61.3% (+23.5%). Our key insight is that such scaffolding of spatial mental models, actively constructing and utilizing internal structured spatial representations with flexible reasoning processes, significantly improves understanding of unobservable space. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21458 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MindCube: Spatial Mental Modeling from Limited Views Wang, Qineng Yin, Baiqiao Zhang, Pingyue Zhang, Jianshu Wang, Kangrui Wang, Zihan Zhang, Jieyu Chandrasegaran, Keshigeyan Liu, Han Krishna, Ranjay Xie, Saining Wu, Jiajun Fei-Fei, Li Li, Manling Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Can Vision-Language Models (VLMs) imagine the full scene from just a few views, like humans do? Humans form spatial mental models naturally, internal representations of unseen space, to reason about layout, perspective, and motion. Our MindCube benchmark with 21,154 questions across 3,268 images exposes this critical gap, where existing VLMs exhibit near-random performance. Using MindCube, we systematically evaluate how well VLMs build robust spatial mental models through representing positions (cognitive mapping), orientations (perspective-taking), and dynamics (mental simulation for "what-if" movements). We then explore three approaches to help approximate spatial mental models in VLMs, focusing on incorporating unseen intermediate views, natural language reasoning chains, and cognitive maps. The significant improvement comes from a synergistic approach, "map-then-reason", that jointly trains the model to first generate a cognitive map and then reason upon it. By training models to reason over these internal maps, we boosted accuracy from 37.8% to 57.8% (+20.0%). Adding reinforcement learning pushed performance even further to 61.3% (+23.5%). Our key insight is that such scaffolding of spatial mental models, actively constructing and utilizing internal structured spatial representations with flexible reasoning processes, significantly improves understanding of unobservable space. |
| title | MindCube: Spatial Mental Modeling from Limited Views |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.21458 |