CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915365428133888 |
|---|---|
| author | Li, Qiming Ye, Zekai Feng, Xiaocheng Zhong, Weihong Qin, Libo Chen, Ruihan Li, Baohang Jiang, Kui Wang, Yaowei Liu, Ting Qin, Bing |
| author_facet | Li, Qiming Ye, Zekai Feng, Xiaocheng Zhong, Weihong Qin, Libo Chen, Ruihan Li, Baohang Jiang, Kui Wang, Yaowei Liu, Ting Qin, Bing |
| contents | Although Large Vision-Language Models (LVLMs) have demonstrated powerful capabilities in interpreting visual information, they frequently produce content that deviates from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly stronger when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-sensitive Attention Intervention (CAI), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern in response to caption queries to enhance LVLMs' visual perception capability. Extensive experimental results across four benchmarks covering both discriminative and generative tasks, demonstrate that CAI achieves state-of-the-art (SOTA) hallucination mitigating performance only with minimal additional inference cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_23590 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models Li, Qiming Ye, Zekai Feng, Xiaocheng Zhong, Weihong Qin, Libo Chen, Ruihan Li, Baohang Jiang, Kui Wang, Yaowei Liu, Ting Qin, Bing Computer Vision and Pattern Recognition Although Large Vision-Language Models (LVLMs) have demonstrated powerful capabilities in interpreting visual information, they frequently produce content that deviates from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly stronger when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-sensitive Attention Intervention (CAI), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern in response to caption queries to enhance LVLMs' visual perception capability. Extensive experimental results across four benchmarks covering both discriminative and generative tasks, demonstrate that CAI achieves state-of-the-art (SOTA) hallucination mitigating performance only with minimal additional inference cost. |
| title | CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.23590 |