Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918446868987904 |
|---|---|
| author | Habba, Eliya Itzhak, Itay Yehudai, Asaf Perlitz, Yotam Bandel, Elron Shmueli-Scheuer, Michal Choshen, Leshem Stanovsky, Gabriel |
| author_facet | Habba, Eliya Itzhak, Itay Yehudai, Asaf Perlitz, Yotam Bandel, Elron Shmueli-Scheuer, Michal Choshen, Leshem Stanovsky, Gabriel |
| contents | The rapid release of both language models and benchmarks makes it increasingly costly to evaluate every model on every dataset. In practice, models are often evaluated on different samples, making scores difficult to compare across studies. To address this, we propose a framework based on multidimensional Item Response Theory (IRT) that uses anchor items to calibrate new benchmarks to the evaluation suite while holding previously calibrated item parameters fixed. Our approach supports a realistic evaluation setting in which datasets are introduced over time and models are evaluated only on the datasets available at the time of evaluation, while a fixed anchor set for each dataset is used so that results from different evaluation periods can be compared directly. In large-scale experiments on more than $400$ models, our framework predicts full-evaluation performance within 2-3 percentage points using only $100$ anchor questions per dataset, with Spearman $ρ\geq 0.9$ for ranking preservation, showing that it is possible to extend benchmark suites over time while preserving score comparability, at a constant evaluation cost per new dataset. Code available at https://github.com/eliyahabba/growing-pains |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_12843 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration Habba, Eliya Itzhak, Itay Yehudai, Asaf Perlitz, Yotam Bandel, Elron Shmueli-Scheuer, Michal Choshen, Leshem Stanovsky, Gabriel Computation and Language The rapid release of both language models and benchmarks makes it increasingly costly to evaluate every model on every dataset. In practice, models are often evaluated on different samples, making scores difficult to compare across studies. To address this, we propose a framework based on multidimensional Item Response Theory (IRT) that uses anchor items to calibrate new benchmarks to the evaluation suite while holding previously calibrated item parameters fixed. Our approach supports a realistic evaluation setting in which datasets are introduced over time and models are evaluated only on the datasets available at the time of evaluation, while a fixed anchor set for each dataset is used so that results from different evaluation periods can be compared directly. In large-scale experiments on more than $400$ models, our framework predicts full-evaluation performance within 2-3 percentage points using only $100$ anchor questions per dataset, with Spearman $ρ\geq 0.9$ for ranking preservation, showing that it is possible to extend benchmark suites over time while preserving score comparability, at a constant evaluation cost per new dataset. Code available at https://github.com/eliyahabba/growing-pains |
| title | Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.12843 |