Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.25334 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914598936903680 |
|---|---|
| author | Zheng, Yufei Zhu, Xuhan Liu, Zide Zhou, Chunpeng Wang, Chenfeng Xu, Yongchao Wang, Yunnan Liu, Jiawei Yu, Pengfei Zhai, Wei Cao, Yang Zha, Zheng-Jun |
| author_facet | Zheng, Yufei Zhu, Xuhan Liu, Zide Zhou, Chunpeng Wang, Chenfeng Xu, Yongchao Wang, Yunnan Liu, Jiawei Yu, Pengfei Zhai, Wei Cao, Yang Zha, Zheng-Jun |
| contents | Spatial understanding of the physical world from 2D visual inputs hinges on two complementary forms of geometric knowledge: holistic 3D structural perception and fine-grained metric scale estimation. Existing multimodal large language models (MLLMs) typically address only one facet, ingesting either depth maps or point clouds as additional model inputs, which incurs substantial computational overhead and inherits the generalization limitations of upstream prediction models. We propose GAMSI, a dual-pathway Geometry-Aware MLLM for Spatial Intelligence that takes only RGB images as input while internalizing both forms of geometric prior within a unified autoregressive backbone. Specifically, we introduce Metric-Structure Decoupled Queries (MSDQ) which employ two groups of learnable queries to respectively extract dense metric signals and sparse structural cues from the shared visual context, with a task-decoupled attention mask further preventing the two pathways from contaminating each other. Building on this, an Expert-Guided Visual Grounding (EVG) module projects the aggregated cues back to frame-level visual features and aligns them with vision foundation models, which serve purely as training-time supervision, rather than as model inputs. We further build a multi-task spatial instruction-tuning dataset (MTS) comprising 152{,}776 samples spanning 13 task types and three visual modalities, consolidated from six public datasets. Trained with a two-stage curriculum, GAMSI achieves state-of-the-art performance on seven spatial intelligence benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25334 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence Zheng, Yufei Zhu, Xuhan Liu, Zide Zhou, Chunpeng Wang, Chenfeng Xu, Yongchao Wang, Yunnan Liu, Jiawei Yu, Pengfei Zhai, Wei Cao, Yang Zha, Zheng-Jun Computer Vision and Pattern Recognition Spatial understanding of the physical world from 2D visual inputs hinges on two complementary forms of geometric knowledge: holistic 3D structural perception and fine-grained metric scale estimation. Existing multimodal large language models (MLLMs) typically address only one facet, ingesting either depth maps or point clouds as additional model inputs, which incurs substantial computational overhead and inherits the generalization limitations of upstream prediction models. We propose GAMSI, a dual-pathway Geometry-Aware MLLM for Spatial Intelligence that takes only RGB images as input while internalizing both forms of geometric prior within a unified autoregressive backbone. Specifically, we introduce Metric-Structure Decoupled Queries (MSDQ) which employ two groups of learnable queries to respectively extract dense metric signals and sparse structural cues from the shared visual context, with a task-decoupled attention mask further preventing the two pathways from contaminating each other. Building on this, an Expert-Guided Visual Grounding (EVG) module projects the aggregated cues back to frame-level visual features and aligns them with vision foundation models, which serve purely as training-time supervision, rather than as model inputs. We further build a multi-task spatial instruction-tuning dataset (MTS) comprising 152{,}776 samples spanning 13 task types and three visual modalities, consolidated from six public datasets. Trained with a two-stage curriculum, GAMSI achieves state-of-the-art performance on seven spatial intelligence benchmarks. |
| title | Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.25334 |