Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916493516603392 |
|---|---|
| author | Qiu, Haoran Mao, Weichao Patke, Archit Cui, Shengkun Jha, Saurabh Wang, Chen Franke, Hubertus Kalbarczyk, Zbigniew T. Başar, Tamer Iyer, Ravishankar K. |
| author_facet | Qiu, Haoran Mao, Weichao Patke, Archit Cui, Shengkun Jha, Saurabh Wang, Chen Franke, Hubertus Kalbarczyk, Zbigniew T. Başar, Tamer Iyer, Ravishankar K. |
| contents | Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_08509 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction Qiu, Haoran Mao, Weichao Patke, Archit Cui, Shengkun Jha, Saurabh Wang, Chen Franke, Hubertus Kalbarczyk, Zbigniew T. Başar, Tamer Iyer, Ravishankar K. Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings. |
| title | Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction |
| topic | Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2404.08509 |