Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.13804 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913558981246976 |
|---|---|
| author | Zhao, Hongyu Li, Ming Sun, Lichao Zhou, Tianyi |
| author_facet | Zhao, Hongyu Li, Ming Sun, Lichao Zhou, Tianyi |
| contents | Evaluating large language models (LLMs) is costly: it requires the generation and examination of LLM outputs on a large-scale benchmark of various tasks. This paper investigates how to efficiently reduce the tasks used to benchmark LLMs without affecting the evaluation quality. Our study reveals that task transferability and relevance provide critical information to identify the most representative subset of tasks via optimizing a facility location function. We propose a practically efficient metric for estimating the transferability between two tasks via in-context learning (ICL). By analyzing the pairwise transferability, we can reduce tasks in a modern LLM benchmark (e.g., MMLU or FLAN) to 5% while inducing only a <4% difference to the evaluation on the original benchmark. Compared to prior works, our method is training-free, gradient-free, and highly efficient requiring ICL only. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13804 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BenTo: Benchmark Task Reduction with In-Context Transferability Zhao, Hongyu Li, Ming Sun, Lichao Zhou, Tianyi Computation and Language Evaluating large language models (LLMs) is costly: it requires the generation and examination of LLM outputs on a large-scale benchmark of various tasks. This paper investigates how to efficiently reduce the tasks used to benchmark LLMs without affecting the evaluation quality. Our study reveals that task transferability and relevance provide critical information to identify the most representative subset of tasks via optimizing a facility location function. We propose a practically efficient metric for estimating the transferability between two tasks via in-context learning (ICL). By analyzing the pairwise transferability, we can reduce tasks in a modern LLM benchmark (e.g., MMLU or FLAN) to 5% while inducing only a <4% difference to the evaluation on the original benchmark. Compared to prior works, our method is training-free, gradient-free, and highly efficient requiring ICL only. |
| title | BenTo: Benchmark Task Reduction with In-Context Transferability |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.13804 |