Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.04504 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915050209411072 |
|---|---|
| author | Guldogan, Ozgur Kunde, Jackson Lee, Kangwook Pedarsani, Ramtin |
| author_facet | Guldogan, Ozgur Kunde, Jackson Lee, Kangwook Pedarsani, Ramtin |
| contents | As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the inference jobs on servers (e.g. GPUs), enabling the system to maximize throughput by allowing multiple requests to be processed in parallel. However, requests often have varying generation lengths, causing resource underutilization, as hardware must wait for the longest-running request in the batch to complete before moving to the next batch. We formalize this problem from a queueing-theoretic perspective, and aim to design a control policy which is throughput-optimal. We propose Multi-Bin Batching, a simple yet effective method that can provably improve LLM inference throughput by grouping requests with similar (predicted) execution times into predetermined bins. Through a combination of theoretical analysis and experiments, including real-world LLM inference scenarios, we demonstrate significant throughput gains compared to standard batching approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04504 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-Bin Batching for Increasing LLM Inference Throughput Guldogan, Ozgur Kunde, Jackson Lee, Kangwook Pedarsani, Ramtin Computation and Language Distributed, Parallel, and Cluster Computing Machine Learning Systems and Control As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the inference jobs on servers (e.g. GPUs), enabling the system to maximize throughput by allowing multiple requests to be processed in parallel. However, requests often have varying generation lengths, causing resource underutilization, as hardware must wait for the longest-running request in the batch to complete before moving to the next batch. We formalize this problem from a queueing-theoretic perspective, and aim to design a control policy which is throughput-optimal. We propose Multi-Bin Batching, a simple yet effective method that can provably improve LLM inference throughput by grouping requests with similar (predicted) execution times into predetermined bins. Through a combination of theoretical analysis and experiments, including real-world LLM inference scenarios, we demonstrate significant throughput gains compared to standard batching approaches. |
| title | Multi-Bin Batching for Increasing LLM Inference Throughput |
| topic | Computation and Language Distributed, Parallel, and Cluster Computing Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2412.04504 |