Towards High-Goodput LLM Serving with Prefill-decode Multiplexing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912886233759744 |
|---|---|
| author | Chen, Yukang Cui, Weihao Zhao, Han Xu, Ziyi Fan, Xiaoze Chen, Xusheng Zhou, Yangjie Sun, Shixuan He, Bingsheng Chen, Quan |
| author_facet | Chen, Yukang Cui, Weihao Zhao, Han Xu, Ziyi Fan, Xiaoze Chen, Xusheng Zhou, Yangjie Sun, Shixuan He, Bingsheng Chen, Quan |
| contents | Large Language Model (LLM) serving must meet stringent Service Level Objectives (SLOs) for both the prefill and decode phases. Some existing solutions disaggregate the two phases, causing potential resource idleness or compute redundancy. Others split the prefill phase into chunks and fuse it with decode iteration, creating a dilemma between SLO compliance and high utilization. To address these issues, an efficient serving system should dynamically adapt compute allocation, decouple compute from memory management, and execute prefill and decode independently. We present MuxWise, an LLM serving framework that adopts a new paradigm, intra-GPU prefill-decode multiplexing, to meet these requirements. To fully exploit the paradigm, MuxWise integrates a bubble-less multiplex engine, a contention-tolerant estimator, and an SLO-aware dispatcher. Evaluation shows that MuxWise improves peak throughput under SLO guarantees by an average of 2.20x (up to 3.06x) over state-of-the-art baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14489 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards High-Goodput LLM Serving with Prefill-decode Multiplexing Chen, Yukang Cui, Weihao Zhao, Han Xu, Ziyi Fan, Xiaoze Chen, Xusheng Zhou, Yangjie Sun, Shixuan He, Bingsheng Chen, Quan Operating Systems Large Language Model (LLM) serving must meet stringent Service Level Objectives (SLOs) for both the prefill and decode phases. Some existing solutions disaggregate the two phases, causing potential resource idleness or compute redundancy. Others split the prefill phase into chunks and fuse it with decode iteration, creating a dilemma between SLO compliance and high utilization. To address these issues, an efficient serving system should dynamically adapt compute allocation, decouple compute from memory management, and execute prefill and decode independently. We present MuxWise, an LLM serving framework that adopts a new paradigm, intra-GPU prefill-decode multiplexing, to meet these requirements. To fully exploit the paradigm, MuxWise integrates a bubble-less multiplex engine, a contention-tolerant estimator, and an SLO-aware dispatcher. Evaluation shows that MuxWise improves peak throughput under SLO guarantees by an average of 2.20x (up to 3.06x) over state-of-the-art baselines. |
| title | Towards High-Goodput LLM Serving with Prefill-decode Multiplexing |
| topic | Operating Systems |
| url | https://arxiv.org/abs/2504.14489 |