CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909017149800448 |
|---|---|
| author | Li, Qiming Ye, Zekai Feng, Xiaocheng Zhong, Weihong Qin, Libo Chen, Ruihan Huang, Lei Li, Baohang Jiang, Kui Wang, Yaowei Liu, Ting Qin, Bing |
| author_facet | Li, Qiming Ye, Zekai Feng, Xiaocheng Zhong, Weihong Qin, Libo Chen, Ruihan Huang, Lei Li, Baohang Jiang, Kui Wang, Yaowei Liu, Ting Qin, Bing |
| contents | Although Large Vision-Language Models (LVLMs) have demonstrated remarkable performance on downstream tasks, they frequently produce contents that deviate from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or decoding strategies which significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly enhanced when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-guided Visual Attention Steering (CAST), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern corresponding to caption queries to enhance LVLMs' visual perception capability. Specifically, we use probing techniques to identify attention heads that are highly sensitive to caption queries and estimate optimized steering directions for their outputs. This steering strengthens LVLM's fine-grained visual perception capabilities, thereby effectively mitigating object hallucination. CAST reduced object hallucination by an average of 6.03% across five widely used LVLMs and five benchmarks including both discriminative and generative tasks, demonstrating state-of-the-art performance while adding little inference cost and preserving other foundational capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_04641 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering Li, Qiming Ye, Zekai Feng, Xiaocheng Zhong, Weihong Qin, Libo Chen, Ruihan Huang, Lei Li, Baohang Jiang, Kui Wang, Yaowei Liu, Ting Qin, Bing Computer Vision and Pattern Recognition Although Large Vision-Language Models (LVLMs) have demonstrated remarkable performance on downstream tasks, they frequently produce contents that deviate from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or decoding strategies which significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly enhanced when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-guided Visual Attention Steering (CAST), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern corresponding to caption queries to enhance LVLMs' visual perception capability. Specifically, we use probing techniques to identify attention heads that are highly sensitive to caption queries and estimate optimized steering directions for their outputs. This steering strengthens LVLM's fine-grained visual perception capabilities, thereby effectively mitigating object hallucination. CAST reduced object hallucination by an average of 6.03% across five widely used LVLMs and five benchmarks including both discriminative and generative tasks, demonstrating state-of-the-art performance while adding little inference cost and preserving other foundational capabilities. |
| title | CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.04641 |