TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915411710181376 |
|---|---|
| author | Liu, Xiaoxuan Park, Jongseok Hu, Langxiang Kwon, Woosuk Li, Zhuohan Zhang, Chen Du, Kuntai Mo, Xiangxi You, Kaichao Cheung, Alvin Deng, Zhijie Stoica, Ion Zhang, Hao |
| author_facet | Liu, Xiaoxuan Park, Jongseok Hu, Langxiang Kwon, Woosuk Li, Zhuohan Zhang, Chen Du, Kuntai Mo, Xiangxi You, Kaichao Cheung, Alvin Deng, Zhijie Stoica, Ion Zhang, Hao |
| contents | Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving. However, in real-world deployments, such inter-request parallelism from batching is often limited by external factors such as low request rates or memory constraints. Recent works focus on intra-request parallelism from speculative decoding as a solution to this problem. Unfortunately, benefits from intra-request parallelism are often fragile, as speculative decoding causes overhead, and speculated tokens may miss. We observe that speculative decoding may degrade LLM serving performance if added naively without tuning to the incoming requests and the speculation method. To alleviate the need for expert tuning and make speculative decoding more robust, we present TurboSpec, a speculation control system that automatically profiles the execution environment and utilizes a feedback-based algorithm to dynamically adjust the amount of intra-request parallelism in LLM serving. TurboSpec predicts "goodput" - the amount of successfully generated tokens - to evaluate and adjust intra-request parallelism amount to that with the highest goodput in runtime. We implement TurboSpec on a real-world LLM serving system vLLM and demonstrate its effectiveness across diverse workloads and hardware configurations, providing consistent performance improvements across all test scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14066 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput Liu, Xiaoxuan Park, Jongseok Hu, Langxiang Kwon, Woosuk Li, Zhuohan Zhang, Chen Du, Kuntai Mo, Xiangxi You, Kaichao Cheung, Alvin Deng, Zhijie Stoica, Ion Zhang, Hao Artificial Intelligence Performance Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving. However, in real-world deployments, such inter-request parallelism from batching is often limited by external factors such as low request rates or memory constraints. Recent works focus on intra-request parallelism from speculative decoding as a solution to this problem. Unfortunately, benefits from intra-request parallelism are often fragile, as speculative decoding causes overhead, and speculated tokens may miss. We observe that speculative decoding may degrade LLM serving performance if added naively without tuning to the incoming requests and the speculation method. To alleviate the need for expert tuning and make speculative decoding more robust, we present TurboSpec, a speculation control system that automatically profiles the execution environment and utilizes a feedback-based algorithm to dynamically adjust the amount of intra-request parallelism in LLM serving. TurboSpec predicts "goodput" - the amount of successfully generated tokens - to evaluate and adjust intra-request parallelism amount to that with the highest goodput in runtime. We implement TurboSpec on a real-world LLM serving system vLLM and demonstrate its effectiveness across diverse workloads and hardware configurations, providing consistent performance improvements across all test scenarios. |
| title | TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput |
| topic | Artificial Intelligence Performance |
| url | https://arxiv.org/abs/2406.14066 |