Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.08147 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916358859522048 |
|---|---|
| author | Jin, Yibo Wang, Tao Lin, Huimin Song, Mingyang Li, Peiyang Ma, Yipeng Shan, Yicheng Yuan, Zhengfan Li, Cailong Sun, Yajing Wu, Tiandeng Chu, Xing Huan, Ruizhi Ma, Li You, Xiao Zhou, Wenting Ye, Yunpeng Liu, Wen Xu, Xiangkun Zhang, Yongsheng Dong, Tiantian Zhu, Jiawei Wang, Zhe Ju, Xijian Song, Jianxun Cheng, Haoliang Li, Xiaojing Ding, Jiandong Guo, Hefei Zhang, Zhengyong |
| author_facet | Jin, Yibo Wang, Tao Lin, Huimin Song, Mingyang Li, Peiyang Ma, Yipeng Shan, Yicheng Yuan, Zhengfan Li, Cailong Sun, Yajing Wu, Tiandeng Chu, Xing Huan, Ruizhi Ma, Li You, Xiao Zhou, Wenting Ye, Yunpeng Liu, Wen Xu, Xiangkun Zhang, Yongsheng Dong, Tiantian Zhu, Jiawei Wang, Zhe Ju, Xijian Song, Jianxun Cheng, Haoliang Li, Xiaojing Ding, Jiandong Guo, Hefei Zhang, Zhengyong |
| contents | Serving disaggregated large language models (LLMs) over tens of thousands of xPU devices (GPUs or NPUs) with reliable performance faces multiple challenges. 1) Ignoring the diversity (various prefixes and tidal requests), treating all the prompts in a mixed pool is inadequate. To facilitate the similarity per scenario and minimize the inner mismatch on P/D (prefill and decoding) processing, fine-grained organization is required, dynamically adjusting P/D ratios for better performance. 2) Due to inaccurate estimation on workload (queue status or maintained connections), the global scheduler easily incurs unnecessary timeouts in prefill. 3) Block-fixed device-to-device (D2D) KVCache transfer over cluster-level RDMA (remote direct memory access) fails to achieve desired D2D utilization as expected. To overcome previous problems, this paper proposes an end-to-end system P/D-Serve, complying with the paradigm of MLOps (machine learning operations), which models end-to-end (E2E) P/D performance and enables: 1) fine-grained P/D organization, mapping the service with RoCE (RDMA over converged ethernet) as needed, to facilitate similar processing and dynamic adjustments on P/D ratios; 2) on-demand forwarding upon rejections for idle prefill, decoupling the scheduler from regular inaccurate reports and local queues, to avoid timeouts in prefill; and 3) efficient KVCache transfer via optimized D2D access. P/D-Serve is implemented upon Ascend and MindSpore, has been deployed over tens of thousands of NPUs for more than eight months in commercial use, and further achieves 60\%, 42\% and 46\% improvements on E2E throughput, time-to-first-token (TTFT) SLO (service level objective) and D2D transfer time. As the E2E system with optimizations, P/D-Serve achieves 6.7x increase on throughput, compared with aggregated LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_08147 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | P/D-Serve: Serving Disaggregated Large Language Model at Scale Jin, Yibo Wang, Tao Lin, Huimin Song, Mingyang Li, Peiyang Ma, Yipeng Shan, Yicheng Yuan, Zhengfan Li, Cailong Sun, Yajing Wu, Tiandeng Chu, Xing Huan, Ruizhi Ma, Li You, Xiao Zhou, Wenting Ye, Yunpeng Liu, Wen Xu, Xiangkun Zhang, Yongsheng Dong, Tiantian Zhu, Jiawei Wang, Zhe Ju, Xijian Song, Jianxun Cheng, Haoliang Li, Xiaojing Ding, Jiandong Guo, Hefei Zhang, Zhengyong Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning Serving disaggregated large language models (LLMs) over tens of thousands of xPU devices (GPUs or NPUs) with reliable performance faces multiple challenges. 1) Ignoring the diversity (various prefixes and tidal requests), treating all the prompts in a mixed pool is inadequate. To facilitate the similarity per scenario and minimize the inner mismatch on P/D (prefill and decoding) processing, fine-grained organization is required, dynamically adjusting P/D ratios for better performance. 2) Due to inaccurate estimation on workload (queue status or maintained connections), the global scheduler easily incurs unnecessary timeouts in prefill. 3) Block-fixed device-to-device (D2D) KVCache transfer over cluster-level RDMA (remote direct memory access) fails to achieve desired D2D utilization as expected. To overcome previous problems, this paper proposes an end-to-end system P/D-Serve, complying with the paradigm of MLOps (machine learning operations), which models end-to-end (E2E) P/D performance and enables: 1) fine-grained P/D organization, mapping the service with RoCE (RDMA over converged ethernet) as needed, to facilitate similar processing and dynamic adjustments on P/D ratios; 2) on-demand forwarding upon rejections for idle prefill, decoupling the scheduler from regular inaccurate reports and local queues, to avoid timeouts in prefill; and 3) efficient KVCache transfer via optimized D2D access. P/D-Serve is implemented upon Ascend and MindSpore, has been deployed over tens of thousands of NPUs for more than eight months in commercial use, and further achieves 60\%, 42\% and 46\% improvements on E2E throughput, time-to-first-token (TTFT) SLO (service level objective) and D2D transfer time. As the E2E system with optimizations, P/D-Serve achieves 6.7x increase on throughput, compared with aggregated LLMs. |
| title | P/D-Serve: Serving Disaggregated Large Language Model at Scale |
| topic | Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2408.08147 |