Sequential Harmful Shift Detection Without Labels
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910749552541696 |
|---|---|
| author | Amoukou, Salim I. Bewley, Tom Mishra, Saumitra Lecue, Freddy Magazzeni, Daniele Veloso, Manuela |
| author_facet | Amoukou, Salim I. Bewley, Tom Mishra, Saumitra Lecue, Freddy Magazzeni, Daniele Veloso, Manuela |
| contents | We introduce a novel approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires no access to ground truth data labels. It builds upon the work of Podkopaev and Ramdas [2022], who address scenarios where labels are available for tracking model errors over time. Our solution extends this framework to work in the absence of labels, by employing a proxy for the true error. This proxy is derived using the predictions of a trained error estimator. Experiments show that our method has high power and false alarm control under various distribution shifts, including covariate and label shifts and natural shifts over geography and time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_12910 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Sequential Harmful Shift Detection Without Labels Amoukou, Salim I. Bewley, Tom Mishra, Saumitra Lecue, Freddy Magazzeni, Daniele Veloso, Manuela Machine Learning We introduce a novel approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires no access to ground truth data labels. It builds upon the work of Podkopaev and Ramdas [2022], who address scenarios where labels are available for tracking model errors over time. Our solution extends this framework to work in the absence of labels, by employing a proxy for the true error. This proxy is derived using the predictions of a trained error estimator. Experiments show that our method has high power and false alarm control under various distribution shifts, including covariate and label shifts and natural shifts over geography and time. |
| title | Sequential Harmful Shift Detection Without Labels |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2412.12910 |