Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908909577437184 |
|---|---|
| author | Gwak, Chanyoung Jeong, Yoonwoo Jeon, Byungwoo Lee, Hyunseok Shin, Jinwoo Cho, Minsu |
| author_facet | Gwak, Chanyoung Jeong, Yoonwoo Jeon, Byungwoo Lee, Hyunseok Shin, Jinwoo Cho, Minsu |
| contents | Precise spatial understanding from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs), as their visual representations are predominantly semantic and lack explicit geometric grounding. While existing approaches augment visual tokens with geometric cues from visual geometry models, their MLLM is still required to implicitly infer the underlying 3D structure of the scene from these augmented tokens, limiting its spatial reasoning capability. To address this issue, we introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images, where each token is grounded in 3D space and possesses both semantic and geometric information. By feeding these tokens into the MLLM, our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks. Code will be made publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_23023 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps Gwak, Chanyoung Jeong, Yoonwoo Jeon, Byungwoo Lee, Hyunseok Shin, Jinwoo Cho, Minsu Computer Vision and Pattern Recognition Precise spatial understanding from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs), as their visual representations are predominantly semantic and lack explicit geometric grounding. While existing approaches augment visual tokens with geometric cues from visual geometry models, their MLLM is still required to implicitly infer the underlying 3D structure of the scene from these augmented tokens, limiting its spatial reasoning capability. To address this issue, we introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images, where each token is grounded in 3D space and possesses both semantic and geometric information. By feeding these tokens into the MLLM, our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks. Code will be made publicly available. |
| title | Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.23023 |