Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914194225364992 |
|---|---|
| author | Jiang, Yanbei Ma, Xueqi Liu, Shu Erfani, Sarah Monazam Liu, Tongliang Bailey, James Lau, Jey Han Ehinger, Krista A. |
| author_facet | Jiang, Yanbei Ma, Xueqi Liu, Shu Erfani, Sarah Monazam Liu, Tongliang Bailey, James Lau, Jey Han Ehinger, Krista A. |
| contents | Despite excelling on multimodal benchmarks, vision-language models (VLMs) largely remain a black box. In this paper, we propose a novel interpretability framework to systematically analyze the internal mechanisms of VLMs, focusing on the functional roles of attention heads in multimodal reasoning. To this end, we introduce CogVision, a dataset that decomposes complex multimodal questions into step-by-step subquestions designed to simulate human reasoning through a chain-of-thought paradigm, with each subquestion associated with specific receptive or cognitive functions such as high-level visual reception and inference. Using a probing-based methodology, we identify attention heads that specialize in these functions and characterize them as functional heads. Our analysis across diverse VLM families reveals that these functional heads are universally sparse, vary in number and distribution across functions, and mediate interactions and hierarchical organization. Furthermore, intervention experiments demonstrate their critical role in multimodal reasoning: removing functional heads leads to performance degradation, while emphasizing them enhances accuracy. These findings provide new insights into the cognitive organization of VLMs and suggest promising directions for designing models with more human-aligned perceptual and reasoning abilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_10300 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules Jiang, Yanbei Ma, Xueqi Liu, Shu Erfani, Sarah Monazam Liu, Tongliang Bailey, James Lau, Jey Han Ehinger, Krista A. Artificial Intelligence Despite excelling on multimodal benchmarks, vision-language models (VLMs) largely remain a black box. In this paper, we propose a novel interpretability framework to systematically analyze the internal mechanisms of VLMs, focusing on the functional roles of attention heads in multimodal reasoning. To this end, we introduce CogVision, a dataset that decomposes complex multimodal questions into step-by-step subquestions designed to simulate human reasoning through a chain-of-thought paradigm, with each subquestion associated with specific receptive or cognitive functions such as high-level visual reception and inference. Using a probing-based methodology, we identify attention heads that specialize in these functions and characterize them as functional heads. Our analysis across diverse VLM families reveals that these functional heads are universally sparse, vary in number and distribution across functions, and mediate interactions and hierarchical organization. Furthermore, intervention experiments demonstrate their critical role in multimodal reasoning: removing functional heads leads to performance degradation, while emphasizing them enhances accuracy. These findings provide new insights into the cognitive organization of VLMs and suggest promising directions for designing models with more human-aligned perceptual and reasoning abilities. |
| title | Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2512.10300 |