Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916368925851648 |
|---|---|
| author | Decker, Thomas Koebler, Alexander Lebacher, Michael Thon, Ingo Tresp, Volker Buettner, Florian |
| author_facet | Decker, Thomas Koebler, Alexander Lebacher, Michael Thon, Ingo Tresp, Volker Buettner, Florian |
| contents | Monitoring and maintaining machine learning models are among the most critical challenges in translating recent advances in the field into real-world applications. However, current monitoring methods lack the capability of provide actionable insights answering the question of why the performance of a particular model really degraded. In this work, we propose a novel approach to explain the behavior of a black-box model under feature shifts by attributing an estimated performance change to interpretable input characteristics. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation (XPE). We analyze the underlying assumptions and demonstrate the superiority of our approach over several baselines on different data sets across various data modalities such as images, audio, and tabular data. We also indicate how the generated results can lead to valuable insights, enabling explanatory model monitoring by revealing potential root causes for model deterioration and guiding toward actionable countermeasures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_13648 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance Decker, Thomas Koebler, Alexander Lebacher, Michael Thon, Ingo Tresp, Volker Buettner, Florian Machine Learning Monitoring and maintaining machine learning models are among the most critical challenges in translating recent advances in the field into real-world applications. However, current monitoring methods lack the capability of provide actionable insights answering the question of why the performance of a particular model really degraded. In this work, we propose a novel approach to explain the behavior of a black-box model under feature shifts by attributing an estimated performance change to interpretable input characteristics. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation (XPE). We analyze the underlying assumptions and demonstrate the superiority of our approach over several baselines on different data sets across various data modalities such as images, audio, and tabular data. We also indicate how the generated results can lead to valuable insights, enabling explanatory model monitoring by revealing potential root causes for model deterioration and guiding toward actionable countermeasures. |
| title | Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2408.13648 |