Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.05056 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912495961112576 |
|---|---|
| author | Dong, Xin Dong, Shichao Wang, Jin Huang, Jing Zhou, Li Sun, Zenghui Jing, Lihua Lan, Jingsong Zhu, Xiaoyong Zheng, Bo |
| author_facet | Dong, Xin Dong, Shichao Wang, Jin Huang, Jing Zhou, Li Sun, Zenghui Jing, Lihua Lan, Jingsong Zhu, Xiaoyong Zheng, Bo |
| contents | Hallucinations in large vision-language models (LVLMs) pose significant challenges for real-world applications, as LVLMs may generate responses that appear plausible yet remain inconsistent with the associated visual content. This issue rarely occurs in human cognition. We argue that this discrepancy arises from humans' ability to effectively leverage multimodal interaction information in data samples. Specifically, humans typically first gather multimodal information, analyze the interactions across modalities for understanding, and then express their understanding through language. Motivated by this observation, we conduct extensive experiments on popular LVLMs and obtained insights that surprisingly reveal human-like, though less pronounced, cognitive behavior of LVLMs on multimodal samples. Building on these findings, we further propose \textbf{INTER}: \textbf{Inter}action Guidance Sampling, a novel training-free algorithm that mitigate hallucinations without requiring additional data. Specifically, INTER explicitly guides LVLMs to effectively reapply their understanding of multimodal interaction information when generating responses, thereby reducing potential hallucinations. On six benchmarks including VQA and image captioning tasks, INTER achieves an average improvement of up to 3.4\% on five LVLMs compared to the state-of-the-art decoding strategy. The code will be released when the paper is accepted. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_05056 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling Dong, Xin Dong, Shichao Wang, Jin Huang, Jing Zhou, Li Sun, Zenghui Jing, Lihua Lan, Jingsong Zhu, Xiaoyong Zheng, Bo Computer Vision and Pattern Recognition Artificial Intelligence Hallucinations in large vision-language models (LVLMs) pose significant challenges for real-world applications, as LVLMs may generate responses that appear plausible yet remain inconsistent with the associated visual content. This issue rarely occurs in human cognition. We argue that this discrepancy arises from humans' ability to effectively leverage multimodal interaction information in data samples. Specifically, humans typically first gather multimodal information, analyze the interactions across modalities for understanding, and then express their understanding through language. Motivated by this observation, we conduct extensive experiments on popular LVLMs and obtained insights that surprisingly reveal human-like, though less pronounced, cognitive behavior of LVLMs on multimodal samples. Building on these findings, we further propose \textbf{INTER}: \textbf{Inter}action Guidance Sampling, a novel training-free algorithm that mitigate hallucinations without requiring additional data. Specifically, INTER explicitly guides LVLMs to effectively reapply their understanding of multimodal interaction information when generating responses, thereby reducing potential hallucinations. On six benchmarks including VQA and image captioning tasks, INTER achieves an average improvement of up to 3.4\% on five LVLMs compared to the state-of-the-art decoding strategy. The code will be released when the paper is accepted. |
| title | INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2507.05056 |