Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908481859092480 |
|---|---|
| author | Wolf, Daniel Hillenhagen, Heiko Taskin, Billurvan Bäuerle, Alex Beer, Meinrad Götz, Michael Ropinski, Timo |
| author_facet | Wolf, Daniel Hillenhagen, Heiko Taskin, Billurvan Bäuerle, Alex Beer, Meinrad Götz, Michael Ropinski, Timo |
| contents | Clinical decision-making relies heavily on understanding relative positions of anatomical structures and anomalies. Therefore, for Vision-Language Models (VLMs) to be applicable in clinical practice, the ability to accurately determine relative positions on medical images is a fundamental prerequisite. Despite its importance, this capability remains highly underexplored. To address this gap, we evaluate the ability of state-of-the-art VLMs, GPT-4o, Llama3.2, Pixtral, and JanusPro, and find that all models fail at this fundamental task. Inspired by successful approaches in computer vision, we investigate whether visual prompts, such as alphanumeric or colored markers placed on anatomical structures, can enhance performance. While these markers provide moderate improvements, results remain significantly lower on medical images compared to observations made on natural images. Our evaluations suggest that, in medical imaging, VLMs rely more on prior anatomical knowledge than on actual image content for answering relative position questions, often leading to incorrect conclusions. To facilitate further research in this area, we introduce the MIRP , Medical Imaging Relative Positioning, benchmark dataset, designed to systematically evaluate the capability to identify relative positions in medical images. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_00549 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images Wolf, Daniel Hillenhagen, Heiko Taskin, Billurvan Bäuerle, Alex Beer, Meinrad Götz, Michael Ropinski, Timo Computer Vision and Pattern Recognition Clinical decision-making relies heavily on understanding relative positions of anatomical structures and anomalies. Therefore, for Vision-Language Models (VLMs) to be applicable in clinical practice, the ability to accurately determine relative positions on medical images is a fundamental prerequisite. Despite its importance, this capability remains highly underexplored. To address this gap, we evaluate the ability of state-of-the-art VLMs, GPT-4o, Llama3.2, Pixtral, and JanusPro, and find that all models fail at this fundamental task. Inspired by successful approaches in computer vision, we investigate whether visual prompts, such as alphanumeric or colored markers placed on anatomical structures, can enhance performance. While these markers provide moderate improvements, results remain significantly lower on medical images compared to observations made on natural images. Our evaluations suggest that, in medical imaging, VLMs rely more on prior anatomical knowledge than on actual image content for answering relative position questions, often leading to incorrect conclusions. To facilitate further research in this area, we introduce the MIRP , Medical Imaging Relative Positioning, benchmark dataset, designed to systematically evaluate the capability to identify relative positions in medical images. |
| title | Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.00549 |