Active Model Selection for Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918158236909568 |
|---|---|
| author | Durmazkeser, Yavuz Okanovic, Patrik Kirsch, Andreas Hoefler, Torsten Gürel, Nezihe Merve |
| author_facet | Durmazkeser, Yavuz Okanovic, Patrik Kirsch, Andreas Hoefler, Torsten Gürel, Nezihe Merve |
| contents | We introduce LLM SELECTOR, the first framework for active model selection of Large Language Models (LLMs). Unlike prior evaluation and benchmarking approaches that rely on fully annotated datasets, LLM SELECTOR efficiently identifies the best LLM with limited annotations. In particular, for any given task, LLM SELECTOR adaptively selects a small set of queries to annotate that are most informative about the best model for the task. To further reduce annotation cost, we leverage a judge-based oracle annotation model. Through extensive experiments on 6 benchmarks with 151 LLMs, we show that LLM SELECTOR reduces annotation costs by up to 59.62% when selecting the best and near-best LLM for the task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09418 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Active Model Selection for Large Language Models Durmazkeser, Yavuz Okanovic, Patrik Kirsch, Andreas Hoefler, Torsten Gürel, Nezihe Merve Computation and Language Machine Learning We introduce LLM SELECTOR, the first framework for active model selection of Large Language Models (LLMs). Unlike prior evaluation and benchmarking approaches that rely on fully annotated datasets, LLM SELECTOR efficiently identifies the best LLM with limited annotations. In particular, for any given task, LLM SELECTOR adaptively selects a small set of queries to annotate that are most informative about the best model for the task. To further reduce annotation cost, we leverage a judge-based oracle annotation model. Through extensive experiments on 6 benchmarks with 151 LLMs, we show that LLM SELECTOR reduces annotation costs by up to 59.62% when selecting the best and near-best LLM for the task. |
| title | Active Model Selection for Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2510.09418 |