Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908590411874304 |
|---|---|
| author | Riera-Marin, Meritxell K, Sikha O Rodriguez-Comas, Julia May, Matthias Stefan Pan, Zhaohong Zhou, Xiang Liang, Xiaokun Erick, Franciskus Xaverius Prenner, Andrea Hemon, Cedric Boussot, Valentin Dillenseger, Jean-Louis Nunes, Jean-Claude Qayyum, Abdul Mazher, Moona Niederer, Steven A Kushibar, Kaisar Martin-Isla, Carlos Radeva, Petia Lekadir, Karim Barfoot, Theodore Herrera, Luis C. Garcia Peraza Glocker, Ben Vercauteren, Tom Gago, Lucas Englemann, Justin Kleiss, Joy-Marie Aubanell, Anton Antolin, Andreu Garcia-Lopez, Javier Ballester, Miguel A. Gonzalez Galdran, Adrian |
| author_facet | Riera-Marin, Meritxell K, Sikha O Rodriguez-Comas, Julia May, Matthias Stefan Pan, Zhaohong Zhou, Xiang Liang, Xiaokun Erick, Franciskus Xaverius Prenner, Andrea Hemon, Cedric Boussot, Valentin Dillenseger, Jean-Louis Nunes, Jean-Claude Qayyum, Abdul Mazher, Moona Niederer, Steven A Kushibar, Kaisar Martin-Isla, Carlos Radeva, Petia Lekadir, Karim Barfoot, Theodore Herrera, Luis C. Garcia Peraza Glocker, Ben Vercauteren, Tom Gago, Lucas Englemann, Justin Kleiss, Joy-Marie Aubanell, Anton Antolin, Andreu Garcia-Lopez, Javier Ballester, Miguel A. Gonzalez Galdran, Adrian |
| contents | Deep learning (DL) has become the dominant approach for medical image segmentation, yet ensuring the reliability and clinical applicability of these models requires addressing key challenges such as annotation variability, calibration, and uncertainty estimation. This is why we created the Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS), which highlights the critical role of multiple annotators in establishing a more comprehensive ground truth, emphasizing that segmentation is inherently subjective and that leveraging inter-annotator variability is essential for robust model evaluation. Seven teams participated in the challenge, submitting a variety of DL models evaluated using metrics such as Dice Similarity Coefficient (DSC), Expected Calibration Error (ECE), and Continuous Ranked Probability Score (CRPS). By incorporating consensus and dissensus ground truth, we assess how DL models handle uncertainty and whether their confidence estimates align with true segmentation performance. Our findings reinforce the importance of well-calibrated models, as better calibration is strongly correlated with the quality of the results. Furthermore, we demonstrate that segmentation models trained on diverse datasets and enriched with pre-trained knowledge exhibit greater robustness, particularly in cases deviating from standard anatomical structures. Notably, the best-performing models achieved high DSC and well-calibrated uncertainty estimates. This work underscores the need for multi-annotator ground truth, thorough calibration assessments, and uncertainty-aware evaluations to develop trustworthy and clinically reliable DL-based medical image segmentation models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_08685 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results Riera-Marin, Meritxell K, Sikha O Rodriguez-Comas, Julia May, Matthias Stefan Pan, Zhaohong Zhou, Xiang Liang, Xiaokun Erick, Franciskus Xaverius Prenner, Andrea Hemon, Cedric Boussot, Valentin Dillenseger, Jean-Louis Nunes, Jean-Claude Qayyum, Abdul Mazher, Moona Niederer, Steven A Kushibar, Kaisar Martin-Isla, Carlos Radeva, Petia Lekadir, Karim Barfoot, Theodore Herrera, Luis C. Garcia Peraza Glocker, Ben Vercauteren, Tom Gago, Lucas Englemann, Justin Kleiss, Joy-Marie Aubanell, Anton Antolin, Andreu Garcia-Lopez, Javier Ballester, Miguel A. Gonzalez Galdran, Adrian Computer Vision and Pattern Recognition Deep learning (DL) has become the dominant approach for medical image segmentation, yet ensuring the reliability and clinical applicability of these models requires addressing key challenges such as annotation variability, calibration, and uncertainty estimation. This is why we created the Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS), which highlights the critical role of multiple annotators in establishing a more comprehensive ground truth, emphasizing that segmentation is inherently subjective and that leveraging inter-annotator variability is essential for robust model evaluation. Seven teams participated in the challenge, submitting a variety of DL models evaluated using metrics such as Dice Similarity Coefficient (DSC), Expected Calibration Error (ECE), and Continuous Ranked Probability Score (CRPS). By incorporating consensus and dissensus ground truth, we assess how DL models handle uncertainty and whether their confidence estimates align with true segmentation performance. Our findings reinforce the importance of well-calibrated models, as better calibration is strongly correlated with the quality of the results. Furthermore, we demonstrate that segmentation models trained on diverse datasets and enriched with pre-trained knowledge exhibit greater robustness, particularly in cases deviating from standard anatomical structures. Notably, the best-performing models achieved high DSC and well-calibrated uncertainty estimates. This work underscores the need for multi-annotator ground truth, thorough calibration assessments, and uncertainty-aware evaluations to develop trustworthy and clinically reliable DL-based medical image segmentation models. |
| title | Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.08685 |