On Speeding Up Language Model Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909513152462848 |
|---|---|
| author | Zhou, Jin Peng Belardi, Christian K. Wu, Ruihan Zhang, Travis Gomes, Carla P. Sun, Wen Weinberger, Kilian Q. |
| author_facet | Zhou, Jin Peng Belardi, Christian K. Wu, Ruihan Zhang, Travis Gomes, Carla P. Sun, Wen Weinberger, Kilian Q. |
| contents | Developing prompt-based methods with Large Language Models (LLMs) requires making numerous decisions, which give rise to a combinatorial search problem over hyper-parameters. This exhaustive evaluation can be time-consuming and costly. In this paper, we propose an $\textit{adaptive}$ approach to explore this space. We are exploiting the fact that often only few samples are needed to identify clearly superior or inferior settings, and that many evaluation tests are highly correlated. We lean on multi-armed bandits to sequentially identify the next (method, validation sample)-pair to evaluate and utilize low-rank matrix factorization to fill in missing evaluations. We carefully assess the efficacy of our approach on several competitive benchmark problems and show that it can identify the top-performing method using only 5-15% of the typical resources -- resulting in 85-95% LLM cost savings. Our code is available at https://github.com/kilian-group/banditeval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_06172 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Speeding Up Language Model Evaluation Zhou, Jin Peng Belardi, Christian K. Wu, Ruihan Zhang, Travis Gomes, Carla P. Sun, Wen Weinberger, Kilian Q. Artificial Intelligence Computation and Language Developing prompt-based methods with Large Language Models (LLMs) requires making numerous decisions, which give rise to a combinatorial search problem over hyper-parameters. This exhaustive evaluation can be time-consuming and costly. In this paper, we propose an $\textit{adaptive}$ approach to explore this space. We are exploiting the fact that often only few samples are needed to identify clearly superior or inferior settings, and that many evaluation tests are highly correlated. We lean on multi-armed bandits to sequentially identify the next (method, validation sample)-pair to evaluate and utilize low-rank matrix factorization to fill in missing evaluations. We carefully assess the efficacy of our approach on several competitive benchmark problems and show that it can identify the top-performing method using only 5-15% of the typical resources -- resulting in 85-95% LLM cost savings. Our code is available at https://github.com/kilian-group/banditeval. |
| title | On Speeding Up Language Model Evaluation |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2407.06172 |