ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915477964455936 |
|---|---|
| author | Qiao, Yifan Anzai, Shu Yu, Shan Ma, Haoran Yang, Shuo Wang, Yang Kim, Miryung Wu, Yongji Zhou, Yang Xing, Jiarong Gonzalez, Joseph E. Stoica, Ion Xu, Harry |
| author_facet | Qiao, Yifan Anzai, Shu Yu, Shan Ma, Haoran Yang, Shuo Wang, Yang Kim, Miryung Wu, Yongji Zhou, Yang Xing, Jiarong Gonzalez, Joseph E. Stoica, Ion Xu, Harry |
| contents | Large language model (LLM) serving demands low latency and high throughput, but high load variability makes it challenging to achieve high GPU utilization. In this paper, we identify a synergetic but overlooked opportunity to co-serve latency-critical online requests alongside latency-tolerant offline tasks such as model benchmarking. While promising, existing serving systems fail to co-serve them efficiently, as their coarse-grained resource management at the request or iteration level cannot harvest millisecond-level GPU idle cycles without introducing interference that violates online latency objectives. ConServe is a new LLM co-serving system that achieves high throughput and strong online latency guarantees by managing resources at finer granularities. ConServe introduces three techniques: (1) a latency-aware token-level scheduler that precisely sizes offline batches and tokens to fit within online latency objectives; (2) sub-iteration, layer-wise preemption that allows offline tasks to yield to online load spikes; and (3) incremental KV cache management that enables preempting and resuming offline requests at near-zero cost. Evaluations with Llama-3.1 and Qwen-2.5 models on real-world workloads show that ConServe delivers an average of 2.2$\times$ higher throughput and reduces online serving tail latency by 2.9$\times$ on average compared to state-of-the-art systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01228 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving Qiao, Yifan Anzai, Shu Yu, Shan Ma, Haoran Yang, Shuo Wang, Yang Kim, Miryung Wu, Yongji Zhou, Yang Xing, Jiarong Gonzalez, Joseph E. Stoica, Ion Xu, Harry Distributed, Parallel, and Cluster Computing Machine Learning Large language model (LLM) serving demands low latency and high throughput, but high load variability makes it challenging to achieve high GPU utilization. In this paper, we identify a synergetic but overlooked opportunity to co-serve latency-critical online requests alongside latency-tolerant offline tasks such as model benchmarking. While promising, existing serving systems fail to co-serve them efficiently, as their coarse-grained resource management at the request or iteration level cannot harvest millisecond-level GPU idle cycles without introducing interference that violates online latency objectives. ConServe is a new LLM co-serving system that achieves high throughput and strong online latency guarantees by managing resources at finer granularities. ConServe introduces three techniques: (1) a latency-aware token-level scheduler that precisely sizes offline batches and tokens to fit within online latency objectives; (2) sub-iteration, layer-wise preemption that allows offline tasks to yield to online load spikes; and (3) incremental KV cache management that enables preempting and resuming offline requests at near-zero cost. Evaluations with Llama-3.1 and Qwen-2.5 models on real-world workloads show that ConServe delivers an average of 2.2$\times$ higher throughput and reduces online serving tail latency by 2.9$\times$ on average compared to state-of-the-art systems. |
| title | ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2410.01228 |