Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909913210421248 |
|---|---|
| author | Li, Cong Yang, Yuzhe Zheng, Xuegui Yang, Qifan Guan, Yijin Zheng, Size Chang, Li-Wen Liu, Shufan Liu, Xin Sun, Guangyu |
| author_facet | Li, Cong Yang, Yuzhe Zheng, Xuegui Yang, Qifan Guan, Yijin Zheng, Size Chang, Li-Wen Liu, Shufan Liu, Xin Sun, Guangyu |
| contents | With the advancement of large language models (LLMs), their context windows have rapidly expanded. To meet diverse demands from varying-length requests in online services, existing state-of-the-art systems tune the sequence parallelism (SP) allocation. However, current dynamic SP allocation lacks flexibility to (1) support stage-specific parallelism requirements in LLM inference, (2) mitigate the global latency degradation from excessive SP allocation, and (3) exploit resource fragments arising from SP size variation.
To tackle this problem, we propose Chunkwise Dynamic Sequence Parallelism (CDSP), a fine-grained parallelism strategy that assigns SP sizes across \textit{intra-request} token segments. Based on CDSP, we build Tetris, an LLM serving system that (1) efficiently integrates CDSP into disaggregated cluster to satisfy parallelism heterogeneity, (2) dynamically regulates SP size expansion based on real-time load conditions, and (3) adaptively explores chunking plans to utilize fragmented resources while meeting per-request demands. Compared with state-of-the-art systems, Tetris achieves up to 4.35$\times$ lower time-to-first-token (TTFT) under max sustainable loads, reduces median time-between-tokens (TBT) by up to 40.1\%, and increases the max request capacity by up to 45\%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_06247 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism Li, Cong Yang, Yuzhe Zheng, Xuegui Yang, Qifan Guan, Yijin Zheng, Size Chang, Li-Wen Liu, Shufan Liu, Xin Sun, Guangyu Distributed, Parallel, and Cluster Computing With the advancement of large language models (LLMs), their context windows have rapidly expanded. To meet diverse demands from varying-length requests in online services, existing state-of-the-art systems tune the sequence parallelism (SP) allocation. However, current dynamic SP allocation lacks flexibility to (1) support stage-specific parallelism requirements in LLM inference, (2) mitigate the global latency degradation from excessive SP allocation, and (3) exploit resource fragments arising from SP size variation. To tackle this problem, we propose Chunkwise Dynamic Sequence Parallelism (CDSP), a fine-grained parallelism strategy that assigns SP sizes across \textit{intra-request} token segments. Based on CDSP, we build Tetris, an LLM serving system that (1) efficiently integrates CDSP into disaggregated cluster to satisfy parallelism heterogeneity, (2) dynamically regulates SP size expansion based on real-time load conditions, and (3) adaptively explores chunking plans to utilize fragmented resources while meeting per-request demands. Compared with state-of-the-art systems, Tetris achieves up to 4.35$\times$ lower time-to-first-token (TTFT) under max sustainable loads, reduces median time-between-tokens (TBT) by up to 40.1\%, and increases the max request capacity by up to 45\%. |
| title | Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2511.06247 |