Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912555940708352 |
|---|---|
| author | Aravindan, Ashwath Vaithinathan Jha, Abha Kulkarni, Mihir |
| author_facet | Aravindan, Ashwath Vaithinathan Jha, Abha Kulkarni, Mihir |
| contents | Vision-Language Models (VLMs) have shown remarkable performance in integrating visual and textual information for tasks such as image captioning and visual question answering. However, these models struggle with compositional generalization and object binding, which limit their ability to handle novel combinations of objects and their attributes. Our work explores the root causes of these failures using mechanistic interpretability techniques. We show evidence that individual neurons in the MLP layers of CLIP's vision encoder represent multiple features, and this "superposition" directly hinders its compositional feature representation which consequently affects compositional reasoning and object binding capabilities. We hope this study will serve as an initial step toward uncovering the mechanistic roots of compositional failures in VLMs. The code and supporting results can be found https://github.com/Mystic-Slice/Do-VLMs-Have-Bad-Eyes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_16652 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability Aravindan, Ashwath Vaithinathan Jha, Abha Kulkarni, Mihir Computer Vision and Pattern Recognition Vision-Language Models (VLMs) have shown remarkable performance in integrating visual and textual information for tasks such as image captioning and visual question answering. However, these models struggle with compositional generalization and object binding, which limit their ability to handle novel combinations of objects and their attributes. Our work explores the root causes of these failures using mechanistic interpretability techniques. We show evidence that individual neurons in the MLP layers of CLIP's vision encoder represent multiple features, and this "superposition" directly hinders its compositional feature representation which consequently affects compositional reasoning and object binding capabilities. We hope this study will serve as an initial step toward uncovering the mechanistic roots of compositional failures in VLMs. The code and supporting results can be found https://github.com/Mystic-Slice/Do-VLMs-Have-Bad-Eyes. |
| title | Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.16652 |