Copertina

Metrics reloaded: Recommendations for image analysis validation

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Maier-Hein, Lena, Reinke, Annika, Godau, Patrick, Tizabi, Minu D., Buettner, Florian, Christodoulou, Evangelia, Glocker, Ben, Isensee, Fabian, Kleesiek, Jens, Kozubek, Michal, Reyes, Mauricio, Riegler, Michael A., Wiesenfarth, Manuel, Kavur, A. Emre, Sudre, Carole H., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Rädsch, Tim, Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Benis, Arriel, Blaschko, Matthew, Cardoso, M. Jorge, Cheplygina, Veronika, Cimini, Beth A., Collins, Gary S., Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Haase, Robert, Hashimoto, Daniel A., Hoffman, Michael M., Huisman, Merel, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Karthikesalingam, Alan, Kenngott, Hannes, Kofler, Florian, Kopp-Schneider, Annette, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Mattson, Peter, Meijering, Erik, Menze, Bjoern, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Petersen, Jens, Rajpoot, Nasir, Rieke, Nicola, Saez-Rodriguez, Julio, Sánchez, Clara I., Shetty, Shravya, van Smeden, Maarten, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, Jäger, Paul F.
Natura:	Preprint
Pubblicazione:	2022
Soggetti:	Computer Vision and Pattern Recognition
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

_version_	1866911782221643776
author	Maier-Hein, Lena Reinke, Annika Godau, Patrick Tizabi, Minu D. Buettner, Florian Christodoulou, Evangelia Glocker, Ben Isensee, Fabian Kleesiek, Jens Kozubek, Michal Reyes, Mauricio Riegler, Michael A. Wiesenfarth, Manuel Kavur, A. Emre Sudre, Carole H. Baumgartner, Michael Eisenmann, Matthias Heckmann-Nötzel, Doreen Rädsch, Tim Acion, Laura Antonelli, Michela Arbel, Tal Bakas, Spyridon Benis, Arriel Blaschko, Matthew Cardoso, M. Jorge Cheplygina, Veronika Cimini, Beth A. Collins, Gary S. Farahani, Keyvan Ferrer, Luciana Galdran, Adrian van Ginneken, Bram Haase, Robert Hashimoto, Daniel A. Hoffman, Michael M. Huisman, Merel Jannin, Pierre Kahn, Charles E. Kainmueller, Dagmar Kainz, Bernhard Karargyris, Alexandros Karthikesalingam, Alan Kenngott, Hannes Kofler, Florian Kopp-Schneider, Annette Kreshuk, Anna Kurc, Tahsin Landman, Bennett A. Litjens, Geert Madani, Amin Maier-Hein, Klaus Martel, Anne L. Mattson, Peter Meijering, Erik Menze, Bjoern Moons, Karel G. M. Müller, Henning Nichyporuk, Brennan Nickel, Felix Petersen, Jens Rajpoot, Nasir Rieke, Nicola Saez-Rodriguez, Julio Sánchez, Clara I. Shetty, Shravya van Smeden, Maarten Summers, Ronald M. Taha, Abdel A. Tiulpin, Aleksei Tsaftaris, Sotirios A. Van Calster, Ben Varoquaux, Gaël Jäger, Paul F.
author_facet	Maier-Hein, Lena Reinke, Annika Godau, Patrick Tizabi, Minu D. Buettner, Florian Christodoulou, Evangelia Glocker, Ben Isensee, Fabian Kleesiek, Jens Kozubek, Michal Reyes, Mauricio Riegler, Michael A. Wiesenfarth, Manuel Kavur, A. Emre Sudre, Carole H. Baumgartner, Michael Eisenmann, Matthias Heckmann-Nötzel, Doreen Rädsch, Tim Acion, Laura Antonelli, Michela Arbel, Tal Bakas, Spyridon Benis, Arriel Blaschko, Matthew Cardoso, M. Jorge Cheplygina, Veronika Cimini, Beth A. Collins, Gary S. Farahani, Keyvan Ferrer, Luciana Galdran, Adrian van Ginneken, Bram Haase, Robert Hashimoto, Daniel A. Hoffman, Michael M. Huisman, Merel Jannin, Pierre Kahn, Charles E. Kainmueller, Dagmar Kainz, Bernhard Karargyris, Alexandros Karthikesalingam, Alan Kenngott, Hannes Kofler, Florian Kopp-Schneider, Annette Kreshuk, Anna Kurc, Tahsin Landman, Bennett A. Litjens, Geert Madani, Amin Maier-Hein, Klaus Martel, Anne L. Mattson, Peter Meijering, Erik Menze, Bjoern Moons, Karel G. M. Müller, Henning Nichyporuk, Brennan Nickel, Felix Petersen, Jens Rajpoot, Nasir Rieke, Nicola Saez-Rodriguez, Julio Sánchez, Clara I. Shetty, Shravya van Smeden, Maarten Summers, Ronald M. Taha, Abdel A. Tiulpin, Aleksei Tsaftaris, Sotirios A. Van Calster, Ben Varoquaux, Gaël Jäger, Paul F.
contents	Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. To overcome this, our large international expert consortium created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as a classification task at image, object or pixel level, namely image-level classification, object detection, semantic segmentation, and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool, which also provides a point of access to explore weaknesses, strengths and specific recommendations for the most common validation metrics. The broad applicability of our framework across domains is demonstrated by an instantiation for various biological and medical image analysis use cases.
format	Preprint
id	arxiv_https___arxiv_org_abs_2206_01653
institution	arXiv
publishDate	2022
record_format	arxiv
spellingShingle	Metrics reloaded: Recommendations for image analysis validation Maier-Hein, Lena Reinke, Annika Godau, Patrick Tizabi, Minu D. Buettner, Florian Christodoulou, Evangelia Glocker, Ben Isensee, Fabian Kleesiek, Jens Kozubek, Michal Reyes, Mauricio Riegler, Michael A. Wiesenfarth, Manuel Kavur, A. Emre Sudre, Carole H. Baumgartner, Michael Eisenmann, Matthias Heckmann-Nötzel, Doreen Rädsch, Tim Acion, Laura Antonelli, Michela Arbel, Tal Bakas, Spyridon Benis, Arriel Blaschko, Matthew Cardoso, M. Jorge Cheplygina, Veronika Cimini, Beth A. Collins, Gary S. Farahani, Keyvan Ferrer, Luciana Galdran, Adrian van Ginneken, Bram Haase, Robert Hashimoto, Daniel A. Hoffman, Michael M. Huisman, Merel Jannin, Pierre Kahn, Charles E. Kainmueller, Dagmar Kainz, Bernhard Karargyris, Alexandros Karthikesalingam, Alan Kenngott, Hannes Kofler, Florian Kopp-Schneider, Annette Kreshuk, Anna Kurc, Tahsin Landman, Bennett A. Litjens, Geert Madani, Amin Maier-Hein, Klaus Martel, Anne L. Mattson, Peter Meijering, Erik Menze, Bjoern Moons, Karel G. M. Müller, Henning Nichyporuk, Brennan Nickel, Felix Petersen, Jens Rajpoot, Nasir Rieke, Nicola Saez-Rodriguez, Julio Sánchez, Clara I. Shetty, Shravya van Smeden, Maarten Summers, Ronald M. Taha, Abdel A. Tiulpin, Aleksei Tsaftaris, Sotirios A. Van Calster, Ben Varoquaux, Gaël Jäger, Paul F. Computer Vision and Pattern Recognition Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. To overcome this, our large international expert consortium created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as a classification task at image, object or pixel level, namely image-level classification, object detection, semantic segmentation, and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool, which also provides a point of access to explore weaknesses, strengths and specific recommendations for the most common validation metrics. The broad applicability of our framework across domains is demonstrated by an instantiation for various biological and medical image analysis use cases.
title	Metrics reloaded: Recommendations for image analysis validation
topic	Computer Vision and Pattern Recognition
url	https://arxiv.org/abs/2206.01653