Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.03475 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910973586046976 |
|---|---|
| author | Liu, Zirui Li, Jiatong Zhuang, Yan Liu, Qi Shen, Shuanghong Ouyang, Jie Cheng, Mingyue Wang, Shijin |
| author_facet | Liu, Zirui Li, Jiatong Zhuang, Yan Liu, Qi Shen, Shuanghong Ouyang, Jie Cheng, Mingyue Wang, Shijin |
| contents | Arena-based evaluation is a fundamental yet significant evaluation paradigm for modern AI models, especially large language models (LLMs). Existing framework based on ELO rating system suffers from the inevitable instability problem due to ranking inconsistency and the lack of attention to the varying abilities of annotators. In this paper, we introduce a novel stable arena framework to address these issues by enhancing the ELO Rating System. Specifically, we replace the iterative update method with a Maximum Likelihood Estimation (MLE) approach, m-ELO, and provide theoretical proof of the consistency and stability of the MLE approach for model ranking. Additionally, we proposed the am-ELO, which modify the Elo Rating's probability function to incorporate annotator abilities, enabling the simultaneous estimation of model scores and annotator reliability. Experiments demonstrate that this method ensures stability, proving that this framework offers a more robust, accurate, and stable evaluation method for LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_03475 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | am-ELO: A Stable Framework for Arena-based LLM Evaluation Liu, Zirui Li, Jiatong Zhuang, Yan Liu, Qi Shen, Shuanghong Ouyang, Jie Cheng, Mingyue Wang, Shijin Artificial Intelligence Machine Learning Arena-based evaluation is a fundamental yet significant evaluation paradigm for modern AI models, especially large language models (LLMs). Existing framework based on ELO rating system suffers from the inevitable instability problem due to ranking inconsistency and the lack of attention to the varying abilities of annotators. In this paper, we introduce a novel stable arena framework to address these issues by enhancing the ELO Rating System. Specifically, we replace the iterative update method with a Maximum Likelihood Estimation (MLE) approach, m-ELO, and provide theoretical proof of the consistency and stability of the MLE approach for model ranking. Additionally, we proposed the am-ELO, which modify the Elo Rating's probability function to incorporate annotator abilities, enabling the simultaneous estimation of model scores and annotator reliability. Experiments demonstrate that this method ensures stability, proving that this framework offers a more robust, accurate, and stable evaluation method for LLMs. |
| title | am-ELO: A Stable Framework for Arena-based LLM Evaluation |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2505.03475 |