Scaling Up Active Testing to Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917101802881024 |
|---|---|
| author | Berrada, Gabrielle Kossen, Jannik Smith, Freddie Bickford Razzak, Muhammed Gal, Yarin Rainforth, Tom |
| author_facet | Berrada, Gabrielle Kossen, Jannik Smith, Freddie Bickford Razzak, Muhammed Gal, Yarin Rainforth, Tom |
| contents | Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving measures that enable active testing to be scaled up to large language models (LLMs). In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without making predictions with the target model. As a result we are able to achieve much more accurate evaluations of LLM performance relative to using randomly acquired data. We additionally introduce a bootstrap estimator of evaluation error, which we show to be a useful indicator of how well active testing is working within a single run. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09093 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Up Active Testing to Large Language Models Berrada, Gabrielle Kossen, Jannik Smith, Freddie Bickford Razzak, Muhammed Gal, Yarin Rainforth, Tom Machine Learning Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving measures that enable active testing to be scaled up to large language models (LLMs). In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without making predictions with the target model. As a result we are able to achieve much more accurate evaluations of LLM performance relative to using randomly acquired data. We additionally introduce a bootstrap estimator of evaluation error, which we show to be a useful indicator of how well active testing is working within a single run. |
| title | Scaling Up Active Testing to Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.09093 |