IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913246738382848 |
|---|---|
| author | Zhu, Lanyun Ji, Deyi Chen, Tianrun Xu, Peng Ye, Jieping Liu, Jun |
| author_facet | Zhu, Lanyun Ji, Deyi Chen, Tianrun Xu, Peng Ye, Jieping Liu, Jun |
| contents | Despite achieving rapid developments and with widespread applications, Large Vision-Language Models (LVLMs) confront a serious challenge of being prone to generating hallucinations. An over-reliance on linguistic priors has been identified as a key factor leading to these hallucinations. In this paper, we propose to alleviate this problem by introducing a novel image-biased decoding (IBD) technique. Our method derives the next-token probability distribution by contrasting predictions from a conventional LVLM with those of an image-biased LVLM, thereby amplifying the correct information highly correlated with image content while mitigating the hallucinatory errors caused by excessive dependence on text. We further conduct a comprehensive statistical analysis to validate the reliability of our method, and design an adaptive adjustment strategy to achieve robust and flexible handling under varying conditions. Experimental results across multiple evaluation metrics verify that our method, despite not requiring additional training data and only with a minimal increase in model parameters, can significantly reduce hallucinations in LVLMs and enhance the truthfulness of the generated response. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_18476 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding Zhu, Lanyun Ji, Deyi Chen, Tianrun Xu, Peng Ye, Jieping Liu, Jun Computer Vision and Pattern Recognition Despite achieving rapid developments and with widespread applications, Large Vision-Language Models (LVLMs) confront a serious challenge of being prone to generating hallucinations. An over-reliance on linguistic priors has been identified as a key factor leading to these hallucinations. In this paper, we propose to alleviate this problem by introducing a novel image-biased decoding (IBD) technique. Our method derives the next-token probability distribution by contrasting predictions from a conventional LVLM with those of an image-biased LVLM, thereby amplifying the correct information highly correlated with image content while mitigating the hallucinatory errors caused by excessive dependence on text. We further conduct a comprehensive statistical analysis to validate the reliability of our method, and design an adaptive adjustment strategy to achieve robust and flexible handling under varying conditions. Experimental results across multiple evaluation metrics verify that our method, despite not requiring additional training data and only with a minimal increase in model parameters, can significantly reduce hallucinations in LVLMs and enhance the truthfulness of the generated response. |
| title | IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.18476 |