Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.17581 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918256223191040 |
|---|---|
| author | Reinke, Annika Christodoulou, Evangelia Sadananda, Sthuthi Kavur, A. Emre Faryna, Khrystyna Schouten, Daan Landman, Bennett A. Sudre, Carole Colliot, Olivier Heller, Nick Loizillon, Sophie Maška, Martin Solal, Maëlys Yazdan-Panah, Arya Bozgo, Vilma Sümer, Ömer de Jong, Siem Fischer, Sophie Kozubek, Michal Rädsch, Tim Hammoud, Nadim Molnár-Gábor, Fruzsina Hicks, Steven Riegler, Michael A. Saha, Anindo Thambawita, Vajira Halvorsen, Pal Jiménez-Sánchez, Amelia Yang, Qingyang Cheplygina, Veronika Bottazzi, Sabrina Seitel, Alexander Bakas, Spyridon Karargyris, Alexandros Venkadesh, Kiran Vaidhya van Ginneken, Bram Maier-Hein, Lena |
| author_facet | Reinke, Annika Christodoulou, Evangelia Sadananda, Sthuthi Kavur, A. Emre Faryna, Khrystyna Schouten, Daan Landman, Bennett A. Sudre, Carole Colliot, Olivier Heller, Nick Loizillon, Sophie Maška, Martin Solal, Maëlys Yazdan-Panah, Arya Bozgo, Vilma Sümer, Ömer de Jong, Siem Fischer, Sophie Kozubek, Michal Rädsch, Tim Hammoud, Nadim Molnár-Gábor, Fruzsina Hicks, Steven Riegler, Michael A. Saha, Anindo Thambawita, Vajira Halvorsen, Pal Jiménez-Sánchez, Amelia Yang, Qingyang Cheplygina, Veronika Bottazzi, Sabrina Seitel, Alexander Bakas, Spyridon Karargyris, Alexandros Venkadesh, Kiran Vaidhya van Ginneken, Bram Maier-Hein, Lena |
| contents | Benchmarking competitions are central to the development of artificial intelligence (AI) in medical imaging, defining performance standards and shaping methodological progress. However, it remains unclear whether these benchmarks provide data that are sufficiently representative, accessible, and reusable to support clinically meaningful AI. In this work, we assess fairness along two complementary dimensions: (1) whether challenge datasets are representative of real-world clinical diversity, and (2) whether they are accessible and legally reusable in line with the FAIR principles. To address this question, we conducted a large-scale systematic study of 241 biomedical image analysis challenges comprising 458 tasks across 19 imaging modalities. Our findings show substantial biases in dataset composition, including geographic location, modality-, and problem type-related biases, indicating that current benchmarks do not adequately reflect real-world clinical diversity. Despite their widespread influence, challenge datasets were frequently constrained by restrictive or ambiguous access conditions, inconsistent or non-compliant licensing practices, and incomplete documentation, limiting reproducibility and long-term reuse. Together, these shortcomings expose foundational fairness limitations in our benchmarking ecosystem and highlight a disconnect between leaderboard success and clinical relevance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_17581 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Medical Imaging AI Competitions Lack Fairness Reinke, Annika Christodoulou, Evangelia Sadananda, Sthuthi Kavur, A. Emre Faryna, Khrystyna Schouten, Daan Landman, Bennett A. Sudre, Carole Colliot, Olivier Heller, Nick Loizillon, Sophie Maška, Martin Solal, Maëlys Yazdan-Panah, Arya Bozgo, Vilma Sümer, Ömer de Jong, Siem Fischer, Sophie Kozubek, Michal Rädsch, Tim Hammoud, Nadim Molnár-Gábor, Fruzsina Hicks, Steven Riegler, Michael A. Saha, Anindo Thambawita, Vajira Halvorsen, Pal Jiménez-Sánchez, Amelia Yang, Qingyang Cheplygina, Veronika Bottazzi, Sabrina Seitel, Alexander Bakas, Spyridon Karargyris, Alexandros Venkadesh, Kiran Vaidhya van Ginneken, Bram Maier-Hein, Lena Computer Vision and Pattern Recognition Benchmarking competitions are central to the development of artificial intelligence (AI) in medical imaging, defining performance standards and shaping methodological progress. However, it remains unclear whether these benchmarks provide data that are sufficiently representative, accessible, and reusable to support clinically meaningful AI. In this work, we assess fairness along two complementary dimensions: (1) whether challenge datasets are representative of real-world clinical diversity, and (2) whether they are accessible and legally reusable in line with the FAIR principles. To address this question, we conducted a large-scale systematic study of 241 biomedical image analysis challenges comprising 458 tasks across 19 imaging modalities. Our findings show substantial biases in dataset composition, including geographic location, modality-, and problem type-related biases, indicating that current benchmarks do not adequately reflect real-world clinical diversity. Despite their widespread influence, challenge datasets were frequently constrained by restrictive or ambiguous access conditions, inconsistent or non-compliant licensing practices, and incomplete documentation, limiting reproducibility and long-term reuse. Together, these shortcomings expose foundational fairness limitations in our benchmarking ecosystem and highlight a disconnect between leaderboard success and clinical relevance. |
| title | Medical Imaging AI Competitions Lack Fairness |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.17581 |