Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909688965103616 |
|---|---|
| author | Chandhok, Shivam Fan, Wan-Cyuan Shwartz, Vered Balasubramanian, Vineeth N Sigal, Leonid |
| author_facet | Chandhok, Shivam Fan, Wan-Cyuan Shwartz, Vered Balasubramanian, Vineeth N Sigal, Leonid |
| contents | Vision-language Models (VLMs) have emerged as general-purpose tools for addressing a variety of complex computer vision problems. Such models have been shown to be highly capable, but, at the same time, lacking some basic visual understanding skills. In this paper, we set out to understand the limitations of SoTA VLMs on fundamental visual tasks by constructing a series of tests that probe which components of design, specifically, may be lacking. Importantly, we go significantly beyond the current benchmarks, which simply measure the final performance of VLM response, by also comparing and contrasting it to the performance of probes trained directly on features obtained from the visual encoder, intermediate vision-language projection and LLM-decoder output. In doing so, we uncover shortcomings in VLMs and make a number of important observations about their capabilities, robustness and how they process visual information. We hope our insights will guide progress in further improving VLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_10442 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities Chandhok, Shivam Fan, Wan-Cyuan Shwartz, Vered Balasubramanian, Vineeth N Sigal, Leonid Machine Learning Artificial Intelligence Vision-language Models (VLMs) have emerged as general-purpose tools for addressing a variety of complex computer vision problems. Such models have been shown to be highly capable, but, at the same time, lacking some basic visual understanding skills. In this paper, we set out to understand the limitations of SoTA VLMs on fundamental visual tasks by constructing a series of tests that probe which components of design, specifically, may be lacking. Importantly, we go significantly beyond the current benchmarks, which simply measure the final performance of VLM response, by also comparing and contrasting it to the performance of probes trained directly on features obtained from the visual encoder, intermediate vision-language projection and LLM-decoder output. In doing so, we uncover shortcomings in VLMs and make a number of important observations about their capabilities, robustness and how they process visual information. We hope our insights will guide progress in further improving VLMs. |
| title | Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2507.10442 |