Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.16061 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913803449401344 |
|---|---|
| author | Khemlani, Sangeet Tran, Tyler Gyory, Nathaniel Harrison, Anthony M. Lawson, Wallace E. Thielstrom, Ravenna Thompson, Hunter Singh, Taaren Trafton, J. Gregory |
| author_facet | Khemlani, Sangeet Tran, Tyler Gyory, Nathaniel Harrison, Anthony M. Lawson, Wallace E. Thielstrom, Ravenna Thompson, Hunter Singh, Taaren Trafton, J. Gregory |
| contents | Vision language models (VLMs) are designed to extract relevant visuospatial information from images. Some research suggests that VLMs can exhibit humanlike scene understanding, while other investigations reveal difficulties in their ability to process relational information. To achieve widespread applicability, VLMs must perform reliably, yielding comparable competence across a wide variety of related tasks. We sought to test how reliable these architectures are at engaging in trivial spatial cognition, e.g., recognizing whether one object is left of another in an uncluttered scene. We developed a benchmark dataset -- TableTest -- whose images depict 3D scenes of objects arranged on a table, and used it to evaluate state-of-the-art VLMs. Results show that performance could be degraded by minor variations of prompts that use logically equivalent descriptions. These analyses suggest limitations in how VLMs may reason about spatial relations in real-world applications. They also reveal novel opportunities for bolstering image caption corpora for more efficient training and testing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_16061 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Vision language models are unreliable at trivial spatial cognition Khemlani, Sangeet Tran, Tyler Gyory, Nathaniel Harrison, Anthony M. Lawson, Wallace E. Thielstrom, Ravenna Thompson, Hunter Singh, Taaren Trafton, J. Gregory Computer Vision and Pattern Recognition Artificial Intelligence Vision language models (VLMs) are designed to extract relevant visuospatial information from images. Some research suggests that VLMs can exhibit humanlike scene understanding, while other investigations reveal difficulties in their ability to process relational information. To achieve widespread applicability, VLMs must perform reliably, yielding comparable competence across a wide variety of related tasks. We sought to test how reliable these architectures are at engaging in trivial spatial cognition, e.g., recognizing whether one object is left of another in an uncluttered scene. We developed a benchmark dataset -- TableTest -- whose images depict 3D scenes of objects arranged on a table, and used it to evaluate state-of-the-art VLMs. Results show that performance could be degraded by minor variations of prompts that use logically equivalent descriptions. These analyses suggest limitations in how VLMs may reason about spatial relations in real-world applications. They also reveal novel opportunities for bolstering image caption corpora for more efficient training and testing. |
| title | Vision language models are unreliable at trivial spatial cognition |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2504.16061 |