Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2508.09035 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918123921211392 |
|---|---|
| author | Jin, Yibo Xu, Yixu Chen, Yue Wang, Chengbin Wang, Tao Huang, Jiaqi Zhang, Rongfei Dong, Yiming Yan, Yuting Cheng, Ke Zhu, Yingjie Wang, Shulan Tang, Qianqian Meng, Shuaishuai Cheng, Guanxin Wang, Ze Miao, Shuyan Wang, Ketao Liu, Wen Yang, Yifan Zhang, Tong Wang, Anran Lu, Chengzhou Dong, Tiantian Zhang, Yongsheng Wang, Zhe Guo, Hefei Liu, Hongjie Lu, Wei Zhang, Zhengyong |
| author_facet | Jin, Yibo Xu, Yixu Chen, Yue Wang, Chengbin Wang, Tao Huang, Jiaqi Zhang, Rongfei Dong, Yiming Yan, Yuting Cheng, Ke Zhu, Yingjie Wang, Shulan Tang, Qianqian Meng, Shuaishuai Cheng, Guanxin Wang, Ze Miao, Shuyan Wang, Ketao Liu, Wen Yang, Yifan Zhang, Tong Wang, Anran Lu, Chengzhou Dong, Tiantian Zhang, Yongsheng Wang, Zhe Guo, Hefei Liu, Hongjie Lu, Wei Zhang, Zhengyong |
| contents | Serving disaggregated large language models has been widely adopted in industrial practice for enhanced performance. However, too many tokens generated in decoding phase, i.e., occupying the resources for a long time, essentially hamper the cloud from achieving a higher throughput. Meanwhile, due to limited on-device resources, the time to first token (TTFT), i.e., the latency of prefill phase, increases dramatically with the growth on prompt length. In order to concur with such a bottleneck on resources, i.e., long occupation in cloud and limited on-device computing capacity, we propose to separate large language model between cloud and devices. That is, the cloud helps a portion of the content for each device, only in its prefill phase. Specifically, after receiving the first token from the cloud, decoupling with its own prefill, the device responds to the user immediately for a lower TTFT. Then, the following tokens from cloud are presented via a speed controller for smoothed TPOT (the time per output token), until the device catches up with the progress. On-device prefill is then amortized using received tokens while the resource usage in cloud is controlled. Moreover, during cloud prefill, the prompt can be refined, using those intermediate data already generated, to further speed up on-device inference. We implement such a scheme P/D-Device, and confirm its superiority over other alternatives. We further propose an algorithm to decide the best settings. Real-trace experiments show that TTFT decreases at least 60%, maximum TPOT is about tens of milliseconds, and cloud throughput increases by up to 15x. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09035 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | P/D-Device: Disaggregated Large Language Model between Cloud and Devices Jin, Yibo Xu, Yixu Chen, Yue Wang, Chengbin Wang, Tao Huang, Jiaqi Zhang, Rongfei Dong, Yiming Yan, Yuting Cheng, Ke Zhu, Yingjie Wang, Shulan Tang, Qianqian Meng, Shuaishuai Cheng, Guanxin Wang, Ze Miao, Shuyan Wang, Ketao Liu, Wen Yang, Yifan Zhang, Tong Wang, Anran Lu, Chengzhou Dong, Tiantian Zhang, Yongsheng Wang, Zhe Guo, Hefei Liu, Hongjie Lu, Wei Zhang, Zhengyong Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning Serving disaggregated large language models has been widely adopted in industrial practice for enhanced performance. However, too many tokens generated in decoding phase, i.e., occupying the resources for a long time, essentially hamper the cloud from achieving a higher throughput. Meanwhile, due to limited on-device resources, the time to first token (TTFT), i.e., the latency of prefill phase, increases dramatically with the growth on prompt length. In order to concur with such a bottleneck on resources, i.e., long occupation in cloud and limited on-device computing capacity, we propose to separate large language model between cloud and devices. That is, the cloud helps a portion of the content for each device, only in its prefill phase. Specifically, after receiving the first token from the cloud, decoupling with its own prefill, the device responds to the user immediately for a lower TTFT. Then, the following tokens from cloud are presented via a speed controller for smoothed TPOT (the time per output token), until the device catches up with the progress. On-device prefill is then amortized using received tokens while the resource usage in cloud is controlled. Moreover, during cloud prefill, the prompt can be refined, using those intermediate data already generated, to further speed up on-device inference. We implement such a scheme P/D-Device, and confirm its superiority over other alternatives. We further propose an algorithm to decide the best settings. Real-trace experiments show that TTFT decreases at least 60%, maximum TPOT is about tens of milliseconds, and cloud throughput increases by up to 15x. |
| title | P/D-Device: Disaggregated Large Language Model between Cloud and Devices |
| topic | Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2508.09035 |