Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.06433 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915188167409664 |
|---|---|
| author | Su, Qidong Zhao, Wei Li, Xin Andoorveedu, Muralidhar Jiang, Chenhao Zhu, Zhanda Song, Kevin Giannoula, Christina Pekhimenko, Gennady |
| author_facet | Su, Qidong Zhao, Wei Li, Xin Andoorveedu, Muralidhar Jiang, Chenhao Zhu, Zhanda Song, Kevin Giannoula, Christina Pekhimenko, Gennady |
| contents | To improve the efficiency of distributed large language model (LLM) inference, various parallelization strategies, such as tensor and pipeline parallelism, have been proposed. However, the distinct computational characteristics inherent in the two stages of LLM inference-prefilling and decoding-render a single static parallelization strategy insufficient for the effective optimization of both stages. In this work, we present Seesaw, an LLM inference engine optimized for throughput-oriented tasks. The key idea behind Seesaw is dynamic model re-sharding, a technique that facilitates the dynamic reconfiguration of parallelization strategies across stages, thereby maximizing throughput at both phases. To mitigate re-sharding overhead and optimize computational efficiency, we employ tiered KV cache buffering and transition-minimizing scheduling. These approaches work synergistically to reduce the overhead caused by frequent stage transitions while ensuring maximum batching efficiency. Our evaluation demonstrates that Seesaw achieves a throughput increase of up to 1.78x (1.36x on average) compared to vLLM, the most widely used state-of-the-art LLM inference engine. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06433 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Seesaw: High-throughput LLM Inference via Model Re-sharding Su, Qidong Zhao, Wei Li, Xin Andoorveedu, Muralidhar Jiang, Chenhao Zhu, Zhanda Song, Kevin Giannoula, Christina Pekhimenko, Gennady Distributed, Parallel, and Cluster Computing Artificial Intelligence To improve the efficiency of distributed large language model (LLM) inference, various parallelization strategies, such as tensor and pipeline parallelism, have been proposed. However, the distinct computational characteristics inherent in the two stages of LLM inference-prefilling and decoding-render a single static parallelization strategy insufficient for the effective optimization of both stages. In this work, we present Seesaw, an LLM inference engine optimized for throughput-oriented tasks. The key idea behind Seesaw is dynamic model re-sharding, a technique that facilitates the dynamic reconfiguration of parallelization strategies across stages, thereby maximizing throughput at both phases. To mitigate re-sharding overhead and optimize computational efficiency, we employ tiered KV cache buffering and transition-minimizing scheduling. These approaches work synergistically to reduce the overhead caused by frequent stage transitions while ensuring maximum batching efficiency. Our evaluation demonstrates that Seesaw achieves a throughput increase of up to 1.78x (1.36x on average) compared to vLLM, the most widely used state-of-the-art LLM inference engine. |
| title | Seesaw: High-throughput LLM Inference via Model Re-sharding |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2503.06433 |