Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.04720 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916883667615744 |
|---|---|
| author | Christodoulou, Evangelia Reinke, Annika Andrè, Pascaline Godau, Patrick Kalinowski, Piotr Houhou, Rola Erkan, Selen Sudre, Carole H. Burgos, Ninon Boutaj, Sofiène Loizillon, Sophie Solal, Maëlys Cheplygina, Veronika Heitz, Charles Kozubek, Michal Antonelli, Michela Rieke, Nicola Gilson, Antoine Mayer, Leon D. Tizabi, Minu D. Cardoso, M. Jorge Simpson, Amber Kopp-Schneider, Annette Varoquaux, Gaël Colliot, Olivier Maier-Hein, Lena |
| author_facet | Christodoulou, Evangelia Reinke, Annika Andrè, Pascaline Godau, Patrick Kalinowski, Piotr Houhou, Rola Erkan, Selen Sudre, Carole H. Burgos, Ninon Boutaj, Sofiène Loizillon, Sophie Solal, Maëlys Cheplygina, Veronika Heitz, Charles Kozubek, Michal Antonelli, Michela Rieke, Nicola Gilson, Antoine Mayer, Leon D. Tizabi, Minu D. Cardoso, M. Jorge Simpson, Amber Kopp-Schneider, Annette Varoquaux, Gaël Colliot, Olivier Maier-Hein, Lena |
| contents | Performance comparisons are fundamental in medical imaging Artificial Intelligence (AI) research, often driving claims of superiority based on relative improvements in common performance metrics. However, such claims frequently rely solely on empirical mean performance. In this paper, we investigate whether newly proposed methods genuinely outperform the state of the art by analyzing a representative cohort of medical imaging papers. We quantify the probability of false claims based on a Bayesian approach that leverages reported results alongside empirically estimated model congruence to estimate whether the relative ranking of methods is likely to have occurred by chance. According to our results, the majority (>80%) of papers claims outperformance when introducing a new method. Our analysis further revealed a high probability (>5%) of false outperformance claims in 86% of classification papers and 53% of segmentation papers. These findings highlight a critical flaw in current benchmarking practices: claims of outperformance in medical imaging AI are frequently unsubstantiated, posing a risk of misdirecting future research efforts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_04720 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | False Promises in Medical Imaging AI? Assessing Validity of Outperformance Claims Christodoulou, Evangelia Reinke, Annika Andrè, Pascaline Godau, Patrick Kalinowski, Piotr Houhou, Rola Erkan, Selen Sudre, Carole H. Burgos, Ninon Boutaj, Sofiène Loizillon, Sophie Solal, Maëlys Cheplygina, Veronika Heitz, Charles Kozubek, Michal Antonelli, Michela Rieke, Nicola Gilson, Antoine Mayer, Leon D. Tizabi, Minu D. Cardoso, M. Jorge Simpson, Amber Kopp-Schneider, Annette Varoquaux, Gaël Colliot, Olivier Maier-Hein, Lena Computer Vision and Pattern Recognition Performance comparisons are fundamental in medical imaging Artificial Intelligence (AI) research, often driving claims of superiority based on relative improvements in common performance metrics. However, such claims frequently rely solely on empirical mean performance. In this paper, we investigate whether newly proposed methods genuinely outperform the state of the art by analyzing a representative cohort of medical imaging papers. We quantify the probability of false claims based on a Bayesian approach that leverages reported results alongside empirically estimated model congruence to estimate whether the relative ranking of methods is likely to have occurred by chance. According to our results, the majority (>80%) of papers claims outperformance when introducing a new method. Our analysis further revealed a high probability (>5%) of false outperformance claims in 86% of classification papers and 53% of segmentation papers. These findings highlight a critical flaw in current benchmarking practices: claims of outperformance in medical imaging AI are frequently unsubstantiated, posing a risk of misdirecting future research efforts. |
| title | False Promises in Medical Imaging AI? Assessing Validity of Outperformance Claims |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.04720 |