Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.13885 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912834824175616 |
|---|---|
| author | Balkır, Esma Pernthaller, Alice Basaldella, Marco Hernández-Orallo, José Collier, Nigel |
| author_facet | Balkır, Esma Pernthaller, Alice Basaldella, Marco Hernández-Orallo, José Collier, Nigel |
| contents | Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked correct/incorrect. We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge) by replacing the Bernoulli response distribution with a heteroskedastic normal distribution. Building on this, we introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items and as cheaply as possible. We validate our method on five benchmarks spanning n-gram-based, embedding-based, and LLM-as-judge metrics. Our method uses 2% of the items while improving ranking correlation by 0.12 τ over random sampling, with 95% accuracy on confident predictions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_13885 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores Balkır, Esma Pernthaller, Alice Basaldella, Marco Hernández-Orallo, José Collier, Nigel Computation and Language Artificial Intelligence Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked correct/incorrect. We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge) by replacing the Bernoulli response distribution with a heteroskedastic normal distribution. Building on this, we introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items and as cheaply as possible. We validate our method on five benchmarks spanning n-gram-based, embedding-based, and LLM-as-judge metrics. Our method uses 2% of the items while improving ranking correlation by 0.12 τ over random sampling, with 95% accuracy on confident predictions. |
| title | Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.13885 |