From Model Explanation to Data Misinterpretation: A Cautionary Analysis of Post Hoc Explainers in Business Research
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910044236283904 |
|---|---|
| author | Wang, Tong Ragodos, Ronilo Feng, Lu Yu Hu |
| author_facet | Wang, Tong Ragodos, Ronilo Feng, Lu Yu Hu |
| contents | Post hoc explainers such as SHAP and LIME are used widely in business research to interpret complex machine learning models. Although they were designed to explain model predictions, there has been an increasing trend in which the explanations they generate are treated as evidence about underlying data relationships. Based on a systematic review of 181 studies, including 56 published in leading journals, we document that this explanation interpretation is widespread and examine its validity. We also evaluate how well post hoc explanations reliably recover the direction and relative importance of features in true data-generating process. We introduce two metrics-direction alignment and strength alignment-and assess SHAP and LIME using simulated data with known ground truth. Although explanations often appear reasonable on average, they exhibit substantial heterogeneity in their alignment and can fail to be aligned even when their predictive accuracy is high. High predictive performance is therefore necessary but insufficient for reliable explanation. We further show that feature correlation and the Rashomon effect (where many equally accurate models rely on different feature attributions) are key drivers of misalignment. Agreement in explanations across such models provides a practical diagnostic of reliability. Overall, our findings caution against using post hoc explainers for hypothesis validation and instead position them as exploratory tools that generate, rather than confirm, substantive insights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16987 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Model Explanation to Data Misinterpretation: A Cautionary Analysis of Post Hoc Explainers in Business Research Wang, Tong Ragodos, Ronilo Feng, Lu Yu Hu Machine Learning Post hoc explainers such as SHAP and LIME are used widely in business research to interpret complex machine learning models. Although they were designed to explain model predictions, there has been an increasing trend in which the explanations they generate are treated as evidence about underlying data relationships. Based on a systematic review of 181 studies, including 56 published in leading journals, we document that this explanation interpretation is widespread and examine its validity. We also evaluate how well post hoc explanations reliably recover the direction and relative importance of features in true data-generating process. We introduce two metrics-direction alignment and strength alignment-and assess SHAP and LIME using simulated data with known ground truth. Although explanations often appear reasonable on average, they exhibit substantial heterogeneity in their alignment and can fail to be aligned even when their predictive accuracy is high. High predictive performance is therefore necessary but insufficient for reliable explanation. We further show that feature correlation and the Rashomon effect (where many equally accurate models rely on different feature attributions) are key drivers of misalignment. Agreement in explanations across such models provides a practical diagnostic of reliability. Overall, our findings caution against using post hoc explainers for hypothesis validation and instead position them as exploratory tools that generate, rather than confirm, substantive insights. |
| title | From Model Explanation to Data Misinterpretation: A Cautionary Analysis of Post Hoc Explainers in Business Research |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2408.16987 |