Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.06655 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916731044233216 |
|---|---|
| author | Chen, Meilin Tian, Jian Ma, Liang Xie, Di Chen, Weijie Zhu, Jiang |
| author_facet | Chen, Meilin Tian, Jian Ma, Liang Xie, Di Chen, Weijie Zhu, Jiang |
| contents | Benchmark contamination has become a significant concern in the LLM evaluation community. Previous Agents-as-an-Evaluator address this issue by involving agents in the generation of questions. Despite their success, the biases in Agents-as-an-Evaluator methods remain largely unexplored. In this paper, we present a theoretical formulation of evaluation bias, providing valuable insights into designing unbiased evaluation protocols. Furthermore, we identify two type of bias in Agents-as-an-Evaluator through carefully designed probing tasks on a minimal Agents-as-an-Evaluator setup. To address these issues, we propose the Unbiased Evaluator, an evaluation protocol that delivers a more comprehensive, unbiased, and interpretable assessment of LLMs.Extensive experiments reveal significant room for improvement in current LLMs. Additionally, we demonstrate that the Unbiased Evaluator not only offers strong evidence of benchmark contamination but also provides interpretable evaluation results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06655 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unbiased Evaluation of Large Language Models from a Causal Perspective Chen, Meilin Tian, Jian Ma, Liang Xie, Di Chen, Weijie Zhu, Jiang Artificial Intelligence Benchmark contamination has become a significant concern in the LLM evaluation community. Previous Agents-as-an-Evaluator address this issue by involving agents in the generation of questions. Despite their success, the biases in Agents-as-an-Evaluator methods remain largely unexplored. In this paper, we present a theoretical formulation of evaluation bias, providing valuable insights into designing unbiased evaluation protocols. Furthermore, we identify two type of bias in Agents-as-an-Evaluator through carefully designed probing tasks on a minimal Agents-as-an-Evaluator setup. To address these issues, we propose the Unbiased Evaluator, an evaluation protocol that delivers a more comprehensive, unbiased, and interpretable assessment of LLMs.Extensive experiments reveal significant room for improvement in current LLMs. Additionally, we demonstrate that the Unbiased Evaluator not only offers strong evidence of benchmark contamination but also provides interpretable evaluation results. |
| title | Unbiased Evaluation of Large Language Models from a Causal Perspective |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2502.06655 |