Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.08459 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911050784309248 |
|---|---|
| author | Xu, Zishan Xie, Shuyi Lv, Qingsong Xiao, Shupei Song, Linlin Wenjuan, Sui Lin, Fan |
| author_facet | Xu, Zishan Xie, Shuyi Lv, Qingsong Xiao, Shupei Song, Linlin Wenjuan, Sui Lin, Fan |
| contents | With the widespread application of Large Language Models (LLMs) in various tasks, the mainstream LLM platforms generate massive user-model interactions daily. In order to efficiently analyze the performance of models and diagnose failures in their answers, it is essential to develop an automated framework to systematically categorize and attribute errors. However, existing evaluation models lack error attribution capability. In this work, we establish a comprehensive Misattribution Framework with 6 primary and 15 secondary categories to facilitate in-depth analysis. Based on this framework, we present AttriData, a dataset specifically designed for error attribution, encompassing misattribution, along with the corresponding scores and feedback. We also propose MisAttributionLLM, a fine-tuned model on AttriData, which is the first general-purpose judge model capable of simultaneously generating score, misattribution, and feedback. Extensive experiments and analyses are conducted to confirm the effectiveness and robustness of our proposed method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_08459 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework Xu, Zishan Xie, Shuyi Lv, Qingsong Xiao, Shupei Song, Linlin Wenjuan, Sui Lin, Fan Computation and Language With the widespread application of Large Language Models (LLMs) in various tasks, the mainstream LLM platforms generate massive user-model interactions daily. In order to efficiently analyze the performance of models and diagnose failures in their answers, it is essential to develop an automated framework to systematically categorize and attribute errors. However, existing evaluation models lack error attribution capability. In this work, we establish a comprehensive Misattribution Framework with 6 primary and 15 secondary categories to facilitate in-depth analysis. Based on this framework, we present AttriData, a dataset specifically designed for error attribution, encompassing misattribution, along with the corresponding scores and feedback. We also propose MisAttributionLLM, a fine-tuned model on AttriData, which is the first general-purpose judge model capable of simultaneously generating score, misattribution, and feedback. Extensive experiments and analyses are conducted to confirm the effectiveness and robustness of our proposed method. |
| title | Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.08459 |