Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908398520369152 |
|---|---|
| author | Tang, Feilong Liu, Chengzhi Xu, Zhongxing Hu, Ming Peng, Zelin Yang, Zhiwei Su, Jionglong Lin, Minquan Peng, Yifan Cheng, Xuelian Razzak, Imran Ge, Zongyuan |
| author_facet | Tang, Feilong Liu, Chengzhi Xu, Zhongxing Hu, Ming Peng, Zelin Yang, Zhiwei Su, Jionglong Lin, Minquan Peng, Yifan Cheng, Xuelian Razzak, Imran Ge, Zongyuan |
| contents | Recent advancements in multimodal large language models (MLLMs) have significantly improved performance in visual question answering. However, they often suffer from hallucinations. In this work, hallucinations are categorized into two main types: initial hallucinations and snowball hallucinations. We argue that adequate contextual information can be extracted directly from the token interaction process. Inspired by causal inference in the decoding strategy, we propose to leverage causal masks to establish information propagation between multimodal tokens. The hypothesis is that insufficient interaction between those tokens may lead the model to rely on outlier tokens, overlooking dense and rich contextual cues. Therefore, we propose to intervene in the propagation process by tackling outlier tokens to enhance in-context inference. With this goal, we present FarSight, a versatile plug-and-play decoding strategy to reduce attention interference from outlier tokens merely by optimizing the causal mask. The heart of our method is effective token propagation. We design an attention register structure within the upper triangular matrix of the causal mask, dynamically allocating attention to capture attention diverted to outlier tokens. Moreover, a positional awareness encoding method with a diminishing masking rate is proposed, allowing the model to attend to further preceding tokens, especially for video sequence tasks. With extensive experiments, FarSight demonstrates significant hallucination-mitigating performance across different MLLMs on both image and video benchmarks, proving its effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16652 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding Tang, Feilong Liu, Chengzhi Xu, Zhongxing Hu, Ming Peng, Zelin Yang, Zhiwei Su, Jionglong Lin, Minquan Peng, Yifan Cheng, Xuelian Razzak, Imran Ge, Zongyuan Computer Vision and Pattern Recognition Machine Learning Recent advancements in multimodal large language models (MLLMs) have significantly improved performance in visual question answering. However, they often suffer from hallucinations. In this work, hallucinations are categorized into two main types: initial hallucinations and snowball hallucinations. We argue that adequate contextual information can be extracted directly from the token interaction process. Inspired by causal inference in the decoding strategy, we propose to leverage causal masks to establish information propagation between multimodal tokens. The hypothesis is that insufficient interaction between those tokens may lead the model to rely on outlier tokens, overlooking dense and rich contextual cues. Therefore, we propose to intervene in the propagation process by tackling outlier tokens to enhance in-context inference. With this goal, we present FarSight, a versatile plug-and-play decoding strategy to reduce attention interference from outlier tokens merely by optimizing the causal mask. The heart of our method is effective token propagation. We design an attention register structure within the upper triangular matrix of the causal mask, dynamically allocating attention to capture attention diverted to outlier tokens. Moreover, a positional awareness encoding method with a diminishing masking rate is proposed, allowing the model to attend to further preceding tokens, especially for video sequence tasks. With extensive experiments, FarSight demonstrates significant hallucination-mitigating performance across different MLLMs on both image and video benchmarks, proving its effectiveness. |
| title | Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2505.16652 |