Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911656826634240 |
|---|---|
| author | Xu, Yang Zhang, Jiefu Sun, Haixiang Zhou, Zihan Cao, Tianyu Aggarwal, Vaneet |
| author_facet | Xu, Yang Zhang, Jiefu Sun, Haixiang Zhou, Zihan Cao, Tianyu Aggarwal, Vaneet |
| contents | Adaptive prompt and program search makes LLM evaluation selection-sensitive. Once benchmark items are reused inside tuning, the observed winner's score need not estimate the fresh-data performance of the full tune-then-deploy procedure. We study inference for this procedure-level target under explicit tuning budgets. We propose SIREN, a selection-aware repeated-split reporting protocol that freezes the post-search shortlist, separates splitwise selection from held-out evaluation, and uses an item-level Gaussian multiplier bootstrap for uncertainty quantification. In a fixed-shortlist regime with smooth stabilized selection, the estimator admits a first-order item-level representation, and the bootstrap yields valid simultaneous inference on a finite budget grid. This supports confidence intervals for procedure-performance curves and pre-specified equal-budget and cross-budget comparisons. Controlled simulations and MMLU-Pro tuning experiments show that winner-based reporting can be optimistic and can change deployment conclusions, while SIREN remains close to the finite-sample reporting target. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_05973 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking Xu, Yang Zhang, Jiefu Sun, Haixiang Zhou, Zihan Cao, Tianyu Aggarwal, Vaneet Machine Learning Artificial Intelligence Applications Adaptive prompt and program search makes LLM evaluation selection-sensitive. Once benchmark items are reused inside tuning, the observed winner's score need not estimate the fresh-data performance of the full tune-then-deploy procedure. We study inference for this procedure-level target under explicit tuning budgets. We propose SIREN, a selection-aware repeated-split reporting protocol that freezes the post-search shortlist, separates splitwise selection from held-out evaluation, and uses an item-level Gaussian multiplier bootstrap for uncertainty quantification. In a fixed-shortlist regime with smooth stabilized selection, the estimator admits a first-order item-level representation, and the bootstrap yields valid simultaneous inference on a finite budget grid. This supports confidence intervals for procedure-performance curves and pre-specified equal-budget and cross-budget comparisons. Controlled simulations and MMLU-Pro tuning experiments show that winner-based reporting can be optimistic and can change deployment conclusions, while SIREN remains close to the finite-sample reporting target. |
| title | Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking |
| topic | Machine Learning Artificial Intelligence Applications |
| url | https://arxiv.org/abs/2605.05973 |