MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916805429166080 |
|---|---|
| author | Lu, Shuaiye Zhou, Linjiang Shi, Xiaochuan |
| author_facet | Lu, Shuaiye Zhou, Linjiang Shi, Xiaochuan |
| contents | Hallucinations in large vision-language models (LVLMs) often stem from the model's sensitivity to image tokens during decoding, as evidenced by attention peaks observed when generating both real and hallucinated entities. To address this, we propose Memory-Driven Sparse Attention Matrix (MDSAM) , a novel training-free approach that dynamically captures and refines the attention allocated to image tokens at each layer. MDSAM memorizes attention patterns and activates updates through alignment during decoding, enhancing focus on relevant image tokens while effectively reducing hallucinations. We evaluate MDSAM on multiple benchmarks for tasks such as image captioning and visual question answering, demonstrating its ability to consistently reduce hallucinations and improve reliability. Compatible with various LVLM architectures, MDSAM highlights its adaptability and effectiveness in mitigating hallucinations without requiring additional training or external tools. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_17664 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation Lu, Shuaiye Zhou, Linjiang Shi, Xiaochuan Computer Vision and Pattern Recognition Hallucinations in large vision-language models (LVLMs) often stem from the model's sensitivity to image tokens during decoding, as evidenced by attention peaks observed when generating both real and hallucinated entities. To address this, we propose Memory-Driven Sparse Attention Matrix (MDSAM) , a novel training-free approach that dynamically captures and refines the attention allocated to image tokens at each layer. MDSAM memorizes attention patterns and activates updates through alignment during decoding, enhancing focus on relevant image tokens while effectively reducing hallucinations. We evaluate MDSAM on multiple benchmarks for tasks such as image captioning and visual question answering, demonstrating its ability to consistently reduce hallucinations and improve reliability. Compatible with various LVLM architectures, MDSAM highlights its adaptability and effectiveness in mitigating hallucinations without requiring additional training or external tools. |
| title | MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.17664 |