Mean Opinion Score as a New Metric for User-Evaluation of XAI Methods
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916339188236288 |
|---|---|
| author | Yu, Hyeon Benois-Pineau, Jenny Bourqui, Romain Giot, Romain Zhukov, Alexey |
| author_facet | Yu, Hyeon Benois-Pineau, Jenny Bourqui, Romain Giot, Romain Zhukov, Alexey |
| contents | This paper investigates the use of Mean Opinion Score (MOS), a common image quality metric, as a user-centric evaluation metric for XAI post-hoc explainers. To measure the MOS, a user experiment is proposed, which has been conducted with explanation maps of intentionally distorted images. Three methods from the family of feature attribution methods - Gradient-weighted Class Activation Mapping (Grad-CAM), Multi-Layered Feature Explanation Method (MLFEM), and Feature Explanation Method (FEM) - are compared with this metric. Additionally, the correlation of this new user-centric metric with automatic metrics is studied via Spearman's rank correlation coefficient. MOS of MLFEM shows the highest correlation with automatic metrics of Insertion Area Under Curve (IAUC) and Deletion Area Under Curve (DAUC). However, the overall correlations are limited, which highlights the lack of consensus between automatic and user-centric metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_20427 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mean Opinion Score as a New Metric for User-Evaluation of XAI Methods Yu, Hyeon Benois-Pineau, Jenny Bourqui, Romain Giot, Romain Zhukov, Alexey Computer Vision and Pattern Recognition Image and Video Processing I.4.7 This paper investigates the use of Mean Opinion Score (MOS), a common image quality metric, as a user-centric evaluation metric for XAI post-hoc explainers. To measure the MOS, a user experiment is proposed, which has been conducted with explanation maps of intentionally distorted images. Three methods from the family of feature attribution methods - Gradient-weighted Class Activation Mapping (Grad-CAM), Multi-Layered Feature Explanation Method (MLFEM), and Feature Explanation Method (FEM) - are compared with this metric. Additionally, the correlation of this new user-centric metric with automatic metrics is studied via Spearman's rank correlation coefficient. MOS of MLFEM shows the highest correlation with automatic metrics of Insertion Area Under Curve (IAUC) and Deletion Area Under Curve (DAUC). However, the overall correlations are limited, which highlights the lack of consensus between automatic and user-centric metrics. |
| title | Mean Opinion Score as a New Metric for User-Evaluation of XAI Methods |
| topic | Computer Vision and Pattern Recognition Image and Video Processing I.4.7 |
| url | https://arxiv.org/abs/2407.20427 |