Incremental Uncertainty-aware Performance Monitoring with Active Labeling Intervention
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912370266210304 |
|---|---|
| author | Koebler, Alexander Decker, Thomas Thon, Ingo Tresp, Volker Buettner, Florian |
| author_facet | Koebler, Alexander Decker, Thomas Thon, Ingo Tresp, Volker Buettner, Florian |
| contents | We study the problem of monitoring machine learning models under gradual distribution shifts, where circumstances change slowly over time, often leading to unnoticed yet significant declines in accuracy. To address this, we propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates performance changes by modeling gradual shifts using optimal transport. In addition, IUPM quantifies the uncertainty in the performance prediction and introduces an active labeling procedure to restore a reliable estimate under a limited labeling budget. Our experiments show that IUPM outperforms existing performance estimation baselines in various gradual shift scenarios and that its uncertainty awareness guides label acquisition more effectively compared to other strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_07023 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Incremental Uncertainty-aware Performance Monitoring with Active Labeling Intervention Koebler, Alexander Decker, Thomas Thon, Ingo Tresp, Volker Buettner, Florian Machine Learning Artificial Intelligence We study the problem of monitoring machine learning models under gradual distribution shifts, where circumstances change slowly over time, often leading to unnoticed yet significant declines in accuracy. To address this, we propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates performance changes by modeling gradual shifts using optimal transport. In addition, IUPM quantifies the uncertainty in the performance prediction and introduces an active labeling procedure to restore a reliable estimate under a limited labeling budget. Our experiments show that IUPM outperforms existing performance estimation baselines in various gradual shift scenarios and that its uncertainty awareness guides label acquisition more effectively compared to other strategies. |
| title | Incremental Uncertainty-aware Performance Monitoring with Active Labeling Intervention |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.07023 |