Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915519548882944 |
|---|---|
| author | Ramaswamy, Ashwin Demeure, Nestor Rrapaj, Ermal |
| author_facet | Ramaswamy, Ashwin Demeure, Nestor Rrapaj, Ermal |
| contents | New large language models (LLMs) are being released every day. Some perform significantly better or worse than expected given their parameter count. Therefore, there is a need for a method to independently evaluate models. The current best way to evaluate a model is to measure its Elo score by comparing it to other models in a series of contests - an expensive operation since humans are ideally required to compare LLM outputs. We observe that when an LLM is asked to judge such contests, the consistency with which it selects a model as the best in a matchup produces a metric that is 91% correlated with its own human-produced Elo score. This provides a simple proxy for Elo scores that can be computed cheaply, without any human data or prior knowledge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23510 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores Ramaswamy, Ashwin Demeure, Nestor Rrapaj, Ermal Artificial Intelligence New large language models (LLMs) are being released every day. Some perform significantly better or worse than expected given their parameter count. Therefore, there is a need for a method to independently evaluate models. The current best way to evaluate a model is to measure its Elo score by comparing it to other models in a series of contests - an expensive operation since humans are ideally required to compare LLM outputs. We observe that when an LLM is asked to judge such contests, the consistency with which it selects a model as the best in a matchup produces a metric that is 91% correlated with its own human-produced Elo score. This provides a simple proxy for Elo scores that can be computed cheaply, without any human data or prior knowledge. |
| title | Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.23510 |