Reallocating Attention Across Layers to Reduce Multimodal Hallucination
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914377065562112 |
|---|---|
| author | Lu, Haolang Chu, Bolun Fu, WeiYe Nan, Guoshun Liu, Junning Pan, Minghui Li, Qiankun Yu, Yi Wang, Hua Wang, Kun |
| author_facet | Lu, Haolang Chu, Bolun Fu, WeiYe Nan, Guoshun Liu, Junning Pan, Minghui Li, Qiankun Yu, Yi Wang, Hua Wang, Kun |
| contents | Multimodal large reasoning models (MLRMs) often suffer from hallucinations that stem not only from insufficient visual grounding but also from imbalanced allocation between perception and reasoning processes. Building upon recent interpretability findings suggesting a staged division of attention across layers, we analyze how this functional misalignment leads to two complementary failure modes: perceptual bias in shallow layers and reasoning drift in deeper layers. To alleviate these issues, we propose Functional Head Identification and Class-Conditioned Rescaling , a lightweight, training-free plugin that identifies perception- and reasoning-oriented heads and adaptively rebalances their layerwise contributions. Our method improves reasoning consistency and visual faithfulness without retraining or any architectural modification. Evaluations across three representative MLRMs and five multimodal reasoning benchmarks show an average 4.2% point gain, with less than 1% additional computation and only 9% baseline latency. Beyond empirical improvements, our study provides an interpretable perspective on regulating cross-layer functional dynamics to enhance the reliability of multimodal reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10285 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reallocating Attention Across Layers to Reduce Multimodal Hallucination Lu, Haolang Chu, Bolun Fu, WeiYe Nan, Guoshun Liu, Junning Pan, Minghui Li, Qiankun Yu, Yi Wang, Hua Wang, Kun Artificial Intelligence Multimodal large reasoning models (MLRMs) often suffer from hallucinations that stem not only from insufficient visual grounding but also from imbalanced allocation between perception and reasoning processes. Building upon recent interpretability findings suggesting a staged division of attention across layers, we analyze how this functional misalignment leads to two complementary failure modes: perceptual bias in shallow layers and reasoning drift in deeper layers. To alleviate these issues, we propose Functional Head Identification and Class-Conditioned Rescaling , a lightweight, training-free plugin that identifies perception- and reasoning-oriented heads and adaptively rebalances their layerwise contributions. Our method improves reasoning consistency and visual faithfulness without retraining or any architectural modification. Evaluations across three representative MLRMs and five multimodal reasoning benchmarks show an average 4.2% point gain, with less than 1% additional computation and only 9% baseline latency. Beyond empirical improvements, our study provides an interpretable perspective on regulating cross-layer functional dynamics to enhance the reliability of multimodal reasoning. |
| title | Reallocating Attention Across Layers to Reduce Multimodal Hallucination |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.10285 |