Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909428406550528 |
|---|---|
| author | Miró-Nicolau, Miquel Jaume-i-Capó, Antoni Moyà-Alcover, Gabriel |
| author_facet | Miró-Nicolau, Miquel Jaume-i-Capó, Antoni Moyà-Alcover, Gabriel |
| contents | The use of eXplainable Artificial Intelligence (XAI) systems has introduced a set of challenges that need resolution. The XAI robustness, or stability, has been one of the goals of the community from its beginning. Multiple authors have proposed evaluating this feature using objective evaluation measures. Nonetheless, many questions remain. With this work, we propose a novel approach to meta-evaluate these metrics, i.e. analyze the correctness of the evaluators. We propose two new tests that allowed us to evaluate two different stability measures: AVG-Sensitiviy and MAX-Senstivity. We tested their reliability in the presence of perfect and robust explanations, generated with a Decision Tree; as well as completely random explanations and prediction. The metrics results showed their incapacity of identify as erroneous the random explanations, highlighting their overall unreliability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10942 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity Miró-Nicolau, Miquel Jaume-i-Capó, Antoni Moyà-Alcover, Gabriel Computer Vision and Pattern Recognition The use of eXplainable Artificial Intelligence (XAI) systems has introduced a set of challenges that need resolution. The XAI robustness, or stability, has been one of the goals of the community from its beginning. Multiple authors have proposed evaluating this feature using objective evaluation measures. Nonetheless, many questions remain. With this work, we propose a novel approach to meta-evaluate these metrics, i.e. analyze the correctness of the evaluators. We propose two new tests that allowed us to evaluate two different stability measures: AVG-Sensitiviy and MAX-Senstivity. We tested their reliability in the presence of perfect and robust explanations, generated with a Decision Tree; as well as completely random explanations and prediction. The metrics results showed their incapacity of identify as erroneous the random explanations, highlighting their overall unreliability. |
| title | Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.10942 |