Re-evaluating Open-ended Evaluation of Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908353424261120 |
|---|---|
| author | Liu, Siqi Gemp, Ian Marris, Luke Piliouras, Georgios Heess, Nicolas Lanctot, Marc |
| author_facet | Liu, Siqi Gemp, Ian Marris, Luke Piliouras, Georgios Heess, Nicolas Lanctot, Marc |
| contents | Evaluation has traditionally focused on ranking candidates for a specific skill. Modern generalist models, such as Large Language Models (LLMs), decidedly outpace this paradigm. Open-ended evaluation systems, where candidate models are compared on user-submitted prompts, have emerged as a popular solution. Despite their many advantages, we show that the current Elo-based rating systems can be susceptible to and even reinforce biases in data, intentional or accidental, due to their sensitivity to redundancies. To address this issue, we propose evaluation as a 3-player game, and introduce novel game-theoretic solution concepts to ensure robustness to redundancy. We show that our method leads to intuitive ratings and provide insights into the competitive landscape of LLM development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20170 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Re-evaluating Open-ended Evaluation of Large Language Models Liu, Siqi Gemp, Ian Marris, Luke Piliouras, Georgios Heess, Nicolas Lanctot, Marc Computer Science and Game Theory Computation and Language Machine Learning Evaluation has traditionally focused on ranking candidates for a specific skill. Modern generalist models, such as Large Language Models (LLMs), decidedly outpace this paradigm. Open-ended evaluation systems, where candidate models are compared on user-submitted prompts, have emerged as a popular solution. Despite their many advantages, we show that the current Elo-based rating systems can be susceptible to and even reinforce biases in data, intentional or accidental, due to their sensitivity to redundancies. To address this issue, we propose evaluation as a 3-player game, and introduce novel game-theoretic solution concepts to ensure robustness to redundancy. We show that our method leads to intuitive ratings and provide insights into the competitive landscape of LLM development. |
| title | Re-evaluating Open-ended Evaluation of Large Language Models |
| topic | Computer Science and Game Theory Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.20170 |