MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909775288074240 |
|---|---|
| author | Daxberger, Erik Wenzel, Nina Griffiths, David Gang, Haiming Lazarow, Justin Kohavi, Gefen Kang, Kai Eichner, Marcin Yang, Yinfei Dehghan, Afshin Grasch, Peter |
| author_facet | Daxberger, Erik Wenzel, Nina Griffiths, David Gang, Haiming Lazarow, Justin Kohavi, Gefen Kang, Kai Eichner, Marcin Yang, Yinfei Dehghan, Afshin Grasch, Peter |
| contents | Multimodal large language models (MLLMs) excel at 2D visual understanding but remain limited in their ability to reason about 3D space. In this work, we leverage large-scale high-quality 3D scene data with open-set annotations to introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data covers diverse spatial tasks including spatial relationship prediction, metric size and distance estimation, and 3D grounding. We show that CA-VQA enables us to train MM-Spatial, a strong generalist MLLM that also achieves state-of-the-art performance on 3D spatial understanding benchmarks, including our own. We show how incorporating metric depth and multi-view inputs (provided in CA-VQA) can further improve 3D understanding, and demonstrate that data alone allows our model to achieve depth perception capabilities comparable to dedicated monocular depth estimation models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_13111 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs Daxberger, Erik Wenzel, Nina Griffiths, David Gang, Haiming Lazarow, Justin Kohavi, Gefen Kang, Kai Eichner, Marcin Yang, Yinfei Dehghan, Afshin Grasch, Peter Computer Vision and Pattern Recognition Computation and Language Machine Learning Multimodal large language models (MLLMs) excel at 2D visual understanding but remain limited in their ability to reason about 3D space. In this work, we leverage large-scale high-quality 3D scene data with open-set annotations to introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data covers diverse spatial tasks including spatial relationship prediction, metric size and distance estimation, and 3D grounding. We show that CA-VQA enables us to train MM-Spatial, a strong generalist MLLM that also achieves state-of-the-art performance on 3D spatial understanding benchmarks, including our own. We show how incorporating metric depth and multi-view inputs (provided in CA-VQA) can further improve 3D understanding, and demonstrate that data alone allows our model to achieve depth perception capabilities comparable to dedicated monocular depth estimation models. |
| title | MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.13111 |