Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916760581570560 |
|---|---|
| author | Yuan, Peiwen Li, Yiwei Feng, Shaoxiong Wang, Xinglin Zhang, Yueqi Shi, Jiayi Tan, Chuyi Pan, Boyuan Hu, Yao Li, Kan |
| author_facet | Yuan, Peiwen Li, Yiwei Feng, Shaoxiong Wang, Xinglin Zhang, Yueqi Shi, Jiayi Tan, Chuyi Pan, Boyuan Hu, Yao Li, Kan |
| contents | LLM-as-Benchmark-Generator methods have been widely studied as a supplement to human annotators for scalable evaluation, while the potential biases within this paradigm remain underexplored. In this work, we systematically define and validate the phenomenon of inflated performance in models evaluated on their self-generated benchmarks, referred to as self-bias, and attribute it to sub-biases arising from question domain, language style, and wrong labels. On this basis, we propose Silencer, a general framework that leverages the heterogeneity between multiple generators at both the sample and benchmark levels to neutralize bias and generate high-quality, self-bias-silenced benchmark. Experimental results across various settings demonstrate that Silencer can suppress self-bias to near zero, significantly improve evaluation effectiveness of the generated benchmark (with an average improvement from 0.655 to 0.833 in Pearson correlation with high-quality human-annotated benchmark), while also exhibiting strong generalizability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20738 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator Yuan, Peiwen Li, Yiwei Feng, Shaoxiong Wang, Xinglin Zhang, Yueqi Shi, Jiayi Tan, Chuyi Pan, Boyuan Hu, Yao Li, Kan Computation and Language LLM-as-Benchmark-Generator methods have been widely studied as a supplement to human annotators for scalable evaluation, while the potential biases within this paradigm remain underexplored. In this work, we systematically define and validate the phenomenon of inflated performance in models evaluated on their self-generated benchmarks, referred to as self-bias, and attribute it to sub-biases arising from question domain, language style, and wrong labels. On this basis, we propose Silencer, a general framework that leverages the heterogeneity between multiple generators at both the sample and benchmark levels to neutralize bias and generate high-quality, self-bias-silenced benchmark. Experimental results across various settings demonstrate that Silencer can suppress self-bias to near zero, significantly improve evaluation effectiveness of the generated benchmark (with an average improvement from 0.655 to 0.833 in Pearson correlation with high-quality human-annotated benchmark), while also exhibiting strong generalizability. |
| title | Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.20738 |