Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2411.03670 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913655314972672 |
|---|---|
| author | Bassi, Pedro R. A. S. Li, Wenxuan Tang, Yucheng Isensee, Fabian Wang, Zifu Chen, Jieneng Chou, Yu-Cheng Kirchhoff, Yannick Rokuss, Maximilian Huang, Ziyan Ye, Jin He, Junjun Wald, Tassilo Ulrich, Constantin Baumgartner, Michael Roy, Saikat Maier-Hein, Klaus H. Jaeger, Paul Ye, Yiwen Xie, Yutong Zhang, Jianpeng Chen, Ziyang Xia, Yong Xing, Zhaohu Zhu, Lei Sadegheih, Yousef Bozorgpour, Afshin Kumari, Pratibha Azad, Reza Merhof, Dorit Shi, Pengcheng Ma, Ting Du, Yuxin Bai, Fan Huang, Tiejun Zhao, Bo Wang, Haonan Li, Xiaomeng Gu, Hanxue Dong, Haoyu Yang, Jichen Mazurowski, Maciej A. Gupta, Saumya Wu, Linshan Zhuang, Jiaxin Chen, Hao Roth, Holger Xu, Daguang Blaschko, Matthew B. Decherchi, Sergio Cavalli, Andrea Yuille, Alan L. Zhou, Zongwei |
| author_facet | Bassi, Pedro R. A. S. Li, Wenxuan Tang, Yucheng Isensee, Fabian Wang, Zifu Chen, Jieneng Chou, Yu-Cheng Kirchhoff, Yannick Rokuss, Maximilian Huang, Ziyan Ye, Jin He, Junjun Wald, Tassilo Ulrich, Constantin Baumgartner, Michael Roy, Saikat Maier-Hein, Klaus H. Jaeger, Paul Ye, Yiwen Xie, Yutong Zhang, Jianpeng Chen, Ziyang Xia, Yong Xing, Zhaohu Zhu, Lei Sadegheih, Yousef Bozorgpour, Afshin Kumari, Pratibha Azad, Reza Merhof, Dorit Shi, Pengcheng Ma, Ting Du, Yuxin Bai, Fan Huang, Tiejun Zhao, Bo Wang, Haonan Li, Xiaomeng Gu, Hanxue Dong, Haoyu Yang, Jichen Mazurowski, Maciej A. Gupta, Saumya Wu, Linshan Zhuang, Jiaxin Chen, Hao Roth, Holger Xu, Daguang Blaschko, Matthew B. Decherchi, Sergio Cavalli, Andrea Yuille, Alan L. Zhou, Zongwei |
| contents | How can we test AI performance? This question seems trivial, but it isn't. Standard benchmarks often have problems such as in-distribution and small-size test sets, oversimplified metrics, unfair comparisons, and short-term outcome pressure. As a consequence, good performance on standard benchmarks does not guarantee success in real-world scenarios. To address these problems, we present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs. This benchmark is based on 5,195 training CT scans from 76 hospitals around the world and 5,903 testing CT scans from 11 additional hospitals. This diverse test set enhances the statistical significance of benchmark results and rigorously evaluates AI algorithms across various out-of-distribution scenarios. We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms on three test sets. In addition, we also evaluated pre-existing AI frameworks--which, differing from algorithms, are more flexible and can support different algorithms--including MONAI from NVIDIA, nnU-Net from DKFZ, and numerous other open-source frameworks. We are committed to expanding this benchmark to encourage more innovation of AI algorithms for the medical domain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_03670 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation? Bassi, Pedro R. A. S. Li, Wenxuan Tang, Yucheng Isensee, Fabian Wang, Zifu Chen, Jieneng Chou, Yu-Cheng Kirchhoff, Yannick Rokuss, Maximilian Huang, Ziyan Ye, Jin He, Junjun Wald, Tassilo Ulrich, Constantin Baumgartner, Michael Roy, Saikat Maier-Hein, Klaus H. Jaeger, Paul Ye, Yiwen Xie, Yutong Zhang, Jianpeng Chen, Ziyang Xia, Yong Xing, Zhaohu Zhu, Lei Sadegheih, Yousef Bozorgpour, Afshin Kumari, Pratibha Azad, Reza Merhof, Dorit Shi, Pengcheng Ma, Ting Du, Yuxin Bai, Fan Huang, Tiejun Zhao, Bo Wang, Haonan Li, Xiaomeng Gu, Hanxue Dong, Haoyu Yang, Jichen Mazurowski, Maciej A. Gupta, Saumya Wu, Linshan Zhuang, Jiaxin Chen, Hao Roth, Holger Xu, Daguang Blaschko, Matthew B. Decherchi, Sergio Cavalli, Andrea Yuille, Alan L. Zhou, Zongwei Computer Vision and Pattern Recognition Artificial Intelligence How can we test AI performance? This question seems trivial, but it isn't. Standard benchmarks often have problems such as in-distribution and small-size test sets, oversimplified metrics, unfair comparisons, and short-term outcome pressure. As a consequence, good performance on standard benchmarks does not guarantee success in real-world scenarios. To address these problems, we present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs. This benchmark is based on 5,195 training CT scans from 76 hospitals around the world and 5,903 testing CT scans from 11 additional hospitals. This diverse test set enhances the statistical significance of benchmark results and rigorously evaluates AI algorithms across various out-of-distribution scenarios. We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms on three test sets. In addition, we also evaluated pre-existing AI frameworks--which, differing from algorithms, are more flexible and can support different algorithms--including MONAI from NVIDIA, nnU-Net from DKFZ, and numerous other open-source frameworks. We are committed to expanding this benchmark to encourage more innovation of AI algorithms for the medical domain. |
| title | Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation? |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2411.03670 |