Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916743668039680 |
|---|---|
| author | Yin, Yanbin Zhou, Kun Wang, Zhen Zhang, Xiangdong Shao, Yifei Hao, Shibo Gu, Yi Liu, Jieyuan Singla, Somanshu Liu, Tianyang Xing, Eric P. Liu, Zhengzhong Jin, Haojian Hu, Zhiting |
| author_facet | Yin, Yanbin Zhou, Kun Wang, Zhen Zhang, Xiangdong Shao, Yifei Hao, Shibo Gu, Yi Liu, Jieyuan Singla, Somanshu Liu, Tianyang Xing, Eric P. Liu, Zhengzhong Jin, Haojian Hu, Zhiting |
| contents | The recent explosion of large language models (LLMs), each with its own general or specialized strengths, makes scalable, reliable benchmarking more urgent than ever. Standard practices nowadays face fundamental trade-offs: closed-ended question-based benchmarks (eg MMLU) struggle with saturation as newer models emerge, while crowd-sourced leaderboards (eg Chatbot Arena) rely on costly and slow human judges. Recently, automated methods (eg LLM-as-a-judge) shed light on the scalability, but risk bias by relying on one or a few "authority" models. To tackle these issues, we propose Decentralized Arena (dearena), a fully automated framework leveraging collective intelligence from all LLMs to evaluate each other. It mitigates single-model judge bias by democratic, pairwise evaluation, and remains efficient at scale through two key components: (1) a coarse-to-fine ranking algorithm for fast incremental insertion of new models with sub-quadratic complexity, and (2) an automatic question selection strategy for the construction of new evaluation dimensions. Across extensive experiments across 66 LLMs, dearena attains up to 97% correlation with human judgements, while significantly reducing the cost. Our code and data will be publicly released on https://github.com/maitrix-org/de-arena. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12808 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models Yin, Yanbin Zhou, Kun Wang, Zhen Zhang, Xiangdong Shao, Yifei Hao, Shibo Gu, Yi Liu, Jieyuan Singla, Somanshu Liu, Tianyang Xing, Eric P. Liu, Zhengzhong Jin, Haojian Hu, Zhiting Computation and Language Machine Learning The recent explosion of large language models (LLMs), each with its own general or specialized strengths, makes scalable, reliable benchmarking more urgent than ever. Standard practices nowadays face fundamental trade-offs: closed-ended question-based benchmarks (eg MMLU) struggle with saturation as newer models emerge, while crowd-sourced leaderboards (eg Chatbot Arena) rely on costly and slow human judges. Recently, automated methods (eg LLM-as-a-judge) shed light on the scalability, but risk bias by relying on one or a few "authority" models. To tackle these issues, we propose Decentralized Arena (dearena), a fully automated framework leveraging collective intelligence from all LLMs to evaluate each other. It mitigates single-model judge bias by democratic, pairwise evaluation, and remains efficient at scale through two key components: (1) a coarse-to-fine ranking algorithm for fast incremental insertion of new models with sub-quadratic complexity, and (2) an automatic question selection strategy for the construction of new evaluation dimensions. Across extensive experiments across 66 LLMs, dearena attains up to 97% correlation with human judgements, while significantly reducing the cost. Our code and data will be publicly released on https://github.com/maitrix-org/de-arena. |
| title | Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.12808 |