Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915369848930304 |
|---|---|
| author | Yang, Jihan Yang, Shusheng Gupta, Anjali W. Han, Rilyn Fei-Fei, Li Xie, Saining |
| author_facet | Yang, Jihan Yang, Shusheng Gupta, Anjali W. Han, Rilyn Fei-Fei, Li Xie, Saining |
| contents | Humans possess the visual-spatial intelligence to remember spaces from sequential visual observations. However, can Multimodal Large Language Models (MLLMs) trained on million-scale video datasets also ``think in space'' from videos? We present a novel video-based visual-spatial intelligence benchmark (VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit competitive - though subhuman - visual-spatial intelligence. We probe models to express how they think in space both linguistically and visually and find that while spatial reasoning capabilities remain the primary bottleneck for MLLMs to reach higher benchmark performance, local world models and spatial awareness do emerge within these models. Notably, prevailing linguistic reasoning techniques (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve performance, whereas explicitly generating cognitive maps during question-answering enhances MLLMs' spatial distance ability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14171 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces Yang, Jihan Yang, Shusheng Gupta, Anjali W. Han, Rilyn Fei-Fei, Li Xie, Saining Computer Vision and Pattern Recognition Humans possess the visual-spatial intelligence to remember spaces from sequential visual observations. However, can Multimodal Large Language Models (MLLMs) trained on million-scale video datasets also ``think in space'' from videos? We present a novel video-based visual-spatial intelligence benchmark (VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit competitive - though subhuman - visual-spatial intelligence. We probe models to express how they think in space both linguistically and visually and find that while spatial reasoning capabilities remain the primary bottleneck for MLLMs to reach higher benchmark performance, local world models and spatial awareness do emerge within these models. Notably, prevailing linguistic reasoning techniques (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve performance, whereas explicitly generating cognitive maps during question-answering enhances MLLMs' spatial distance ability. |
| title | Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.14171 |