Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.22480 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914997158805504 |
|---|---|
| author | Zhang, Kexun Zhou, Shang Wang, Danqing Wang, William Yang Li, Lei |
| author_facet | Zhang, Kexun Zhou, Shang Wang, Danqing Wang, William Yang Li, Lei |
| contents | Sampling is a basic operation in many inference-time algorithms of large language models (LLMs). To scale up inference efficiently with a limited compute, it is crucial to find an optimal allocation for sample compute budgets: Which sampling configurations (model, temperature, language, etc.) do we use? How many samples do we generate in each configuration? We formulate these choices as a learning problem and propose OSCA, an algorithm that Optimizes Sample Compute Allocation by finding an optimal mix of different inference configurations. Our experiments show that with our learned mixed allocation, we can achieve accuracy better than the best single configuration with 128x less compute on code generation and 25x less compute on 4 reasoning tasks. OSCA is also shown to be effective in agentic workflows beyond single-turn tasks, achieving a better accuracy on SWE-Bench with 3x less compute than the default configuration. Our code and generations are released at https://github.com/LeiLiLab/OSCA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_22480 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scaling LLM Inference with Optimized Sample Compute Allocation Zhang, Kexun Zhou, Shang Wang, Danqing Wang, William Yang Li, Lei Computation and Language Artificial Intelligence Sampling is a basic operation in many inference-time algorithms of large language models (LLMs). To scale up inference efficiently with a limited compute, it is crucial to find an optimal allocation for sample compute budgets: Which sampling configurations (model, temperature, language, etc.) do we use? How many samples do we generate in each configuration? We formulate these choices as a learning problem and propose OSCA, an algorithm that Optimizes Sample Compute Allocation by finding an optimal mix of different inference configurations. Our experiments show that with our learned mixed allocation, we can achieve accuracy better than the best single configuration with 128x less compute on code generation and 25x less compute on 4 reasoning tasks. OSCA is also shown to be effective in agentic workflows beyond single-turn tasks, achieving a better accuracy on SWE-Bench with 3x less compute than the default configuration. Our code and generations are released at https://github.com/LeiLiLab/OSCA. |
| title | Scaling LLM Inference with Optimized Sample Compute Allocation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.22480 |