Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915246799585280 |
|---|---|
| author | Campbell, Declan Rane, Sunayana Giallanza, Tyler De Sabbata, Nicolò Ghods, Kia Joshi, Amogh Ku, Alexander Frankland, Steven M. Griffiths, Thomas L. Cohen, Jonathan D. Webb, Taylor W. |
| author_facet | Campbell, Declan Rane, Sunayana Giallanza, Tyler De Sabbata, Nicolò Ghods, Kia Joshi, Amogh Ku, Alexander Frankland, Steven M. Griffiths, Thomas L. Cohen, Jonathan D. Webb, Taylor W. |
| contents | Recent work has documented striking heterogeneity in the performance of state-of-the-art vision language models (VLMs), including both multimodal language models and text-to-image models. These models are able to describe and generate a diverse array of complex, naturalistic images, yet they exhibit surprising failures on basic multi-object reasoning tasks -- such as counting, localization, and simple forms of visual analogy -- that humans perform with near perfect accuracy. To better understand this puzzling pattern of successes and failures, we turn to theoretical accounts of the binding problem in cognitive science and neuroscience, a fundamental problem that arises when a shared set of representational resources must be used to represent distinct entities (e.g., to represent multiple objects in an image), necessitating the use of serial processing to avoid interference. We find that many of the puzzling failures of state-of-the-art VLMs can be explained as arising due to the binding problem, and that these failure modes are strikingly similar to the limitations exhibited by rapid, feedforward processing in the human brain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_00238 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem Campbell, Declan Rane, Sunayana Giallanza, Tyler De Sabbata, Nicolò Ghods, Kia Joshi, Amogh Ku, Alexander Frankland, Steven M. Griffiths, Thomas L. Cohen, Jonathan D. Webb, Taylor W. Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Neurons and Cognition Recent work has documented striking heterogeneity in the performance of state-of-the-art vision language models (VLMs), including both multimodal language models and text-to-image models. These models are able to describe and generate a diverse array of complex, naturalistic images, yet they exhibit surprising failures on basic multi-object reasoning tasks -- such as counting, localization, and simple forms of visual analogy -- that humans perform with near perfect accuracy. To better understand this puzzling pattern of successes and failures, we turn to theoretical accounts of the binding problem in cognitive science and neuroscience, a fundamental problem that arises when a shared set of representational resources must be used to represent distinct entities (e.g., to represent multiple objects in an image), necessitating the use of serial processing to avoid interference. We find that many of the puzzling failures of state-of-the-art VLMs can be explained as arising due to the binding problem, and that these failure modes are strikingly similar to the limitations exhibited by rapid, feedforward processing in the human brain. |
| title | Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem |
| topic | Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Neurons and Cognition |
| url | https://arxiv.org/abs/2411.00238 |