Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2510.22593 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918171526561792 |
|---|---|
| author | Loi, Dario Muià, Elena Maria Siciliano, Federico Trappolini, Giovanni Crisà, Vincenzo Kruger, Peter Silvestri, Fabrizio |
| author_facet | Loi, Dario Muià, Elena Maria Siciliano, Federico Trappolini, Giovanni Crisà, Vincenzo Kruger, Peter Silvestri, Fabrizio |
| contents | We present AutoBench, a fully automated and self-sustaining framework for evaluating Large Language Models (LLMs) through reciprocal peer assessment. This paper provides a rigorous scientific validation of the AutoBench methodology, originally developed as an open-source project by eZecute S.R.L.. Unlike static benchmarks that suffer from test-set contamination and limited adaptability, AutoBench dynamically generates novel evaluation tasks while models alternately serve as question generators, contestants, and judges across diverse domains. An iterative weighting mechanism amplifies the influence of consistently reliable evaluators, aggregating peer judgments into consensus-based rankings that reflect collective model agreement. Our experiments demonstrate strong correlations with established benchmarks including MMLU-Pro and GPQA (respectively 78\% and 63\%), validating this peer-driven evaluation paradigm. The multi-judge design significantly outperforms single-judge baselines, confirming that distributed evaluation produces more robust and human-consistent assessments. AutoBench offers a scalable, contamination-resistant alternative to static benchmarks for the continuous evaluation of evolving language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22593 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment Loi, Dario Muià, Elena Maria Siciliano, Federico Trappolini, Giovanni Crisà, Vincenzo Kruger, Peter Silvestri, Fabrizio Computation and Language Artificial Intelligence I.2.7; I.2.11; H.3.4; D.2.8 We present AutoBench, a fully automated and self-sustaining framework for evaluating Large Language Models (LLMs) through reciprocal peer assessment. This paper provides a rigorous scientific validation of the AutoBench methodology, originally developed as an open-source project by eZecute S.R.L.. Unlike static benchmarks that suffer from test-set contamination and limited adaptability, AutoBench dynamically generates novel evaluation tasks while models alternately serve as question generators, contestants, and judges across diverse domains. An iterative weighting mechanism amplifies the influence of consistently reliable evaluators, aggregating peer judgments into consensus-based rankings that reflect collective model agreement. Our experiments demonstrate strong correlations with established benchmarks including MMLU-Pro and GPQA (respectively 78\% and 63\%), validating this peer-driven evaluation paradigm. The multi-judge design significantly outperforms single-judge baselines, confirming that distributed evaluation produces more robust and human-consistent assessments. AutoBench offers a scalable, contamination-resistant alternative to static benchmarks for the continuous evaluation of evolving language models. |
| title | AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment |
| topic | Computation and Language Artificial Intelligence I.2.7; I.2.11; H.3.4; D.2.8 |
| url | https://arxiv.org/abs/2510.22593 |