BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909403588853760 |
|---|---|
| author | Zhao, Yilong Yang, Shuo Zhu, Kan Zheng, Lianmin Kasikci, Baris Zhou, Yang Xing, Jiarong Stoica, Ion |
| author_facet | Zhao, Yilong Yang, Shuo Zhu, Kan Zheng, Lianmin Kasikci, Baris Zhou, Yang Xing, Jiarong Stoica, Ion |
| contents | Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applications. Meanwhile, recent progress in model capability and modality makes requests more diverse in compute and memory demands, creating unique opportunities for throughput improvement by resource overlapping. However, a request schedule that maximizes resource overlapping can conflict with the schedule that maximizes prefix sharing, a widely-used performance optimization, causing sub-optimal inference throughput. We present BlendServe, a system that maximizes resource utilization of offline batch inference by combining the benefits of resource overlapping and prefix sharing using a resource-aware prefix tree. BlendServe exploits the relaxed latency requirements in offline batch inference to reorder and overlap requests with varied resource demands while ensuring high prefix sharing. We evaluate BlendServe on a variety of synthetic multi-modal workloads and show that it provides up to $1.44\times$ throughput boost compared to widely-used industry standards, vLLM and SGLang. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16102 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching Zhao, Yilong Yang, Shuo Zhu, Kan Zheng, Lianmin Kasikci, Baris Zhou, Yang Xing, Jiarong Stoica, Ion Machine Learning Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applications. Meanwhile, recent progress in model capability and modality makes requests more diverse in compute and memory demands, creating unique opportunities for throughput improvement by resource overlapping. However, a request schedule that maximizes resource overlapping can conflict with the schedule that maximizes prefix sharing, a widely-used performance optimization, causing sub-optimal inference throughput. We present BlendServe, a system that maximizes resource utilization of offline batch inference by combining the benefits of resource overlapping and prefix sharing using a resource-aware prefix tree. BlendServe exploits the relaxed latency requirements in offline batch inference to reorder and overlap requests with varied resource demands while ensuring high prefix sharing. We evaluate BlendServe on a variety of synthetic multi-modal workloads and show that it provides up to $1.44\times$ throughput boost compared to widely-used industry standards, vLLM and SGLang. |
| title | BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.16102 |