Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911761951621120 |
|---|---|
| author | Hu, Cunchen Huang, Heyang Xu, Liangliang Chen, Xusheng Xu, Jiang Chen, Shuang Feng, Hao Wang, Chenxi Wang, Sa Bao, Yungang Sun, Ninghui Shan, Yizhou |
| author_facet | Hu, Cunchen Huang, Heyang Xu, Liangliang Chen, Xusheng Xu, Jiang Chen, Shuang Feng, Hao Wang, Chenxi Wang, Sa Bao, Yungang Sun, Ninghui Shan, Yizhou |
| contents | Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computationsaturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_11181 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads Hu, Cunchen Huang, Heyang Xu, Liangliang Chen, Xusheng Xu, Jiang Chen, Shuang Feng, Hao Wang, Chenxi Wang, Sa Bao, Yungang Sun, Ninghui Shan, Yizhou Distributed, Parallel, and Cluster Computing Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computationsaturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively. |
| title | Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2401.11181 |