Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.12693 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910993785815040 |
|---|---|
| author | Zhang, Wenyu Ng, Wei En Ma, Lixin Wang, Yuwen Zhao, Junqi Koenecke, Allison Li, Boyang Wang, Lu |
| author_facet | Zhang, Wenyu Ng, Wei En Ma, Lixin Wang, Yuwen Zhao, Junqi Koenecke, Allison Li, Boyang Wang, Lu |
| contents | Current vision-language models may grasp basic spatial cues and simple directions (e.g. left, right, front, back), but struggle with the multi-dimensional spatial reasoning necessary for human-like understanding and real-world applications. To address this gap, we develop SPHERE (Spatial Perception and Hierarchical Evaluation of REasoning), a hierarchical evaluation framework supported by a new human-annotated dataset. SPHERE systematically probes models across increasing levels of complexity, from fundamental skills to multi-skill integration and high-level reasoning that combines spatial, visual, and logical understanding. Benchmark evaluation of state-of-the-art models reveals significant deficiencies, especially in reasoning about distance and proximity, understanding both egocentric and allocentric perspectives, and applying spatial logic in physical contexts. These findings expose critical blind spots in existing models and underscore the need for more advanced spatial reasoning techniques, driving the development of vision-language models that align more closely with human spatial cognition. The SPHERE benchmark is available at https://github.com/zwenyu/SPHERE-VLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_12693 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation Zhang, Wenyu Ng, Wei En Ma, Lixin Wang, Yuwen Zhao, Junqi Koenecke, Allison Li, Boyang Wang, Lu Computer Vision and Pattern Recognition Artificial Intelligence Current vision-language models may grasp basic spatial cues and simple directions (e.g. left, right, front, back), but struggle with the multi-dimensional spatial reasoning necessary for human-like understanding and real-world applications. To address this gap, we develop SPHERE (Spatial Perception and Hierarchical Evaluation of REasoning), a hierarchical evaluation framework supported by a new human-annotated dataset. SPHERE systematically probes models across increasing levels of complexity, from fundamental skills to multi-skill integration and high-level reasoning that combines spatial, visual, and logical understanding. Benchmark evaluation of state-of-the-art models reveals significant deficiencies, especially in reasoning about distance and proximity, understanding both egocentric and allocentric perspectives, and applying spatial logic in physical contexts. These findings expose critical blind spots in existing models and underscore the need for more advanced spatial reasoning techniques, driving the development of vision-language models that align more closely with human spatial cognition. The SPHERE benchmark is available at https://github.com/zwenyu/SPHERE-VLM. |
| title | SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2412.12693 |