Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.19164 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909503325208576 |
|---|---|
| author | Zhang, Kejia Tao, Keda Tang, Jiasheng Wang, Huan |
| author_facet | Zhang, Kejia Tao, Keda Tang, Jiasheng Wang, Huan |
| contents | Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel visual adversarial perturbation (VAP) method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_19164 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs Zhang, Kejia Tao, Keda Tang, Jiasheng Wang, Huan Computer Vision and Pattern Recognition Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel visual adversarial perturbation (VAP) method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations. |
| title | Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.19164 |