Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908684453412864 |
|---|---|
| author | Mahdavi, Zahra Khodakaramimaghsoud, Zahra Khaloo, Hooman Taleshani, Sina Bakhshandeh Hashemi, Erfan Kaleybar, Javad Mirzapour Manzari, Omid Nejati |
| author_facet | Mahdavi, Zahra Khodakaramimaghsoud, Zahra Khaloo, Hooman Taleshani, Sina Bakhshandeh Hashemi, Erfan Kaleybar, Javad Mirzapour Manzari, Omid Nejati |
| contents | Large vision-language models (LVLMs) are now central to healthcare applications such as medical visual question answering and imaging report generation. Yet, these models remain vulnerable to hallucination outputs that appear plausible but are in fact incorrect. In the natural image domain, several decoding strategies have been proposed to mitigate hallucinations by reinforcing visual evidence, but most rely on secondary decoding or rollback procedures that substantially slow inference. Moreover, existing solutions are often domain-specific and may introduce misalignment between modalities or between generated and ground-truth content. We introduce Med-VCD, a sparse visual-contrastive decoding method that mitigates hallucinations in medical LVLMs without the time overhead of secondary decoding. Med-VCD incorporates a novel token-sparsification strategy that selects visually informed tokens on the fly, trimming redundancy while retaining critical visual context and thus balancing efficiency with reliability. Evaluations on eight medical datasets, spanning ophthalmology, radiology, and pathology tasks in visual question answering, report generation, and dedicated hallucination benchmarks, show that Med-VCD raises factual accuracy by an average of 13\% and improves hallucination accuracy by 6\% relative to baseline medical LVLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_01922 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding Mahdavi, Zahra Khodakaramimaghsoud, Zahra Khaloo, Hooman Taleshani, Sina Bakhshandeh Hashemi, Erfan Kaleybar, Javad Mirzapour Manzari, Omid Nejati Computer Vision and Pattern Recognition Large vision-language models (LVLMs) are now central to healthcare applications such as medical visual question answering and imaging report generation. Yet, these models remain vulnerable to hallucination outputs that appear plausible but are in fact incorrect. In the natural image domain, several decoding strategies have been proposed to mitigate hallucinations by reinforcing visual evidence, but most rely on secondary decoding or rollback procedures that substantially slow inference. Moreover, existing solutions are often domain-specific and may introduce misalignment between modalities or between generated and ground-truth content. We introduce Med-VCD, a sparse visual-contrastive decoding method that mitigates hallucinations in medical LVLMs without the time overhead of secondary decoding. Med-VCD incorporates a novel token-sparsification strategy that selects visually informed tokens on the fly, trimming redundancy while retaining critical visual context and thus balancing efficiency with reliability. Evaluations on eight medical datasets, spanning ophthalmology, radiology, and pathology tasks in visual question answering, report generation, and dedicated hallucination benchmarks, show that Med-VCD raises factual accuracy by an average of 13\% and improves hallucination accuracy by 6\% relative to baseline medical LVLMs. |
| title | Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.01922 |