Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.06989 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929309120200704 |
|---|---|
| author | Baker, Ryan S. Bosch, Nigel Hutt, Stephen Zambrano, Andres F. Bowers, Alex J. |
| author_facet | Baker, Ryan S. Bosch, Nigel Hutt, Stephen Zambrano, Andres F. Bowers, Alex J. |
| contents | Recently, ACM FAccT published an article by Kwegyir-Aggrey and colleagues (2023), critiquing the use of AUC ROC in predictive analytics in several domains. In this article, we offer a critique of that article. Specifically, we highlight technical inaccuracies in that paper's comparison of metrics, mis-specification of the interpretation and goals of AUC ROC, the article's use of the accuracy metric as a gold standard for comparison to AUC ROC, and the article's application of critiques solely to AUC ROC for concerns that would apply to the use of any metric. We conclude with a re-framing of the very valid concerns raised in that article, and discuss how the use of AUC ROC can remain a valid and appropriate practice in a well-informed predictive analytics approach taking those concerns into account. We conclude by discussing the combined use of multiple metrics, including machine learning bias metrics, and AUC ROC's place in such an approach. Like broccoli, AUC ROC is healthy, but also like broccoli, researchers and practitioners in our field shouldn't eat a diet of only AUC ROC. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06989 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Fixing the Right Problems in Predictive Analytics: AUC Is Not the Problem Baker, Ryan S. Bosch, Nigel Hutt, Stephen Zambrano, Andres F. Bowers, Alex J. Machine Learning Recently, ACM FAccT published an article by Kwegyir-Aggrey and colleagues (2023), critiquing the use of AUC ROC in predictive analytics in several domains. In this article, we offer a critique of that article. Specifically, we highlight technical inaccuracies in that paper's comparison of metrics, mis-specification of the interpretation and goals of AUC ROC, the article's use of the accuracy metric as a gold standard for comparison to AUC ROC, and the article's application of critiques solely to AUC ROC for concerns that would apply to the use of any metric. We conclude with a re-framing of the very valid concerns raised in that article, and discuss how the use of AUC ROC can remain a valid and appropriate practice in a well-informed predictive analytics approach taking those concerns into account. We conclude by discussing the combined use of multiple metrics, including machine learning bias metrics, and AUC ROC's place in such an approach. Like broccoli, AUC ROC is healthy, but also like broccoli, researchers and practitioners in our field shouldn't eat a diet of only AUC ROC. |
| title | On Fixing the Right Problems in Predictive Analytics: AUC Is Not the Problem |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2404.06989 |