Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2511.19220 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914172276572160 |
|---|---|
| author | Felizzi, Federico Riccomi, Olivia Ferramola, Michele Causio, Francesco Andrea Del Medico, Manuel De Vita, Vittorio De Mori, Lorenzo Piscitelli, Alessandra Risuleo, Pietro Eric Castaniti, Bianca Destro Cristiano, Antonio Longo, Alessia De Angelis, Luigi Vassalli, Mariapia Di Pumpo, Marcello |
| author_facet | Felizzi, Federico Riccomi, Olivia Ferramola, Michele Causio, Francesco Andrea Del Medico, Manuel De Vita, Vittorio De Mori, Lorenzo Piscitelli, Alessandra Risuleo, Pietro Eric Castaniti, Bianca Destro Cristiano, Antonio Longo, Alessia De Angelis, Luigi Vassalli, Mariapia Di Pumpo, Marcello |
| contents | Large vision language models (VLMs) have achieved impressive performance on medical visual question answering benchmarks, yet their reliance on visual information remains unclear. We investigate whether frontier VLMs demonstrate genuine visual grounding when answering Italian medical questions by testing four state-of-the-art models: Claude Sonnet 4.5, GPT-4o, GPT-5-mini, and Gemini 2.0 flash exp. Using 60 questions from the EuropeMedQA Italian dataset that explicitly require image interpretation, we substitute correct medical images with blank placeholders to test whether models truly integrate visual and textual information. Our results reveal striking variability in visual dependency: GPT-4o shows the strongest visual grounding with a 27.9pp accuracy drop (83.2% [74.6%, 91.7%] to 55.3% [44.1%, 66.6%]), while GPT-5-mini, Gemini, and Claude maintain high accuracy with modest drops of 8.5pp, 2.4pp, and 5.6pp respectively. Analysis of model-generated reasoning reveals confident explanations for fabricated visual interpretations across all models, suggesting varying degrees of reliance on textual shortcuts versus genuine visual analysis. These findings highlight critical differences in model robustness and the need for rigorous evaluation before clinical deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_19220 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering Felizzi, Federico Riccomi, Olivia Ferramola, Michele Causio, Francesco Andrea Del Medico, Manuel De Vita, Vittorio De Mori, Lorenzo Piscitelli, Alessandra Risuleo, Pietro Eric Castaniti, Bianca Destro Cristiano, Antonio Longo, Alessia De Angelis, Luigi Vassalli, Mariapia Di Pumpo, Marcello Computer Vision and Pattern Recognition Artificial Intelligence Large vision language models (VLMs) have achieved impressive performance on medical visual question answering benchmarks, yet their reliance on visual information remains unclear. We investigate whether frontier VLMs demonstrate genuine visual grounding when answering Italian medical questions by testing four state-of-the-art models: Claude Sonnet 4.5, GPT-4o, GPT-5-mini, and Gemini 2.0 flash exp. Using 60 questions from the EuropeMedQA Italian dataset that explicitly require image interpretation, we substitute correct medical images with blank placeholders to test whether models truly integrate visual and textual information. Our results reveal striking variability in visual dependency: GPT-4o shows the strongest visual grounding with a 27.9pp accuracy drop (83.2% [74.6%, 91.7%] to 55.3% [44.1%, 66.6%]), while GPT-5-mini, Gemini, and Claude maintain high accuracy with modest drops of 8.5pp, 2.4pp, and 5.6pp respectively. Analysis of model-generated reasoning reveals confident explanations for fabricated visual interpretations across all models, suggesting varying degrees of reliance on textual shortcuts versus genuine visual analysis. These findings highlight critical differences in model robustness and the need for rigorous evaluation before clinical deployment. |
| title | Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2511.19220 |