PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909011663650816 |
|---|---|
| author | Zhang, Hongbin Wei, Taosheng Jiang, Jiazhi Yan, Hui Du, Jiangsu Chen, Zhiguang |
| author_facet | Zhang, Hongbin Wei, Taosheng Jiang, Jiazhi Yan, Hui Du, Jiangsu Chen, Zhiguang |
| contents | Offline LLM inference seeks to maximize request processing under fixed budgets, making commodity GPU servers a promising choice. However, prior work typically considers offloading and parallelism in isolation, resulting in suboptimal performance. In this paper, we propose PipeMax, a high-throughput LLM inference system that integrates pipeline parallelism with offloading to overcome interconnect and memory constraints on GPU servers. Particularly, pipeline parallelism naturally incurs low communication overhead and keeps only one batch active on each GPU at a time, which enables offloading the KV cache of inactive batches. By coordinating computation with offloading data movement, PipeMax effectively expands GPU memory capacity and sustains large-batch execution. Experiments show that PipeMax achieves up to 2.51x higher throughput than vLLM, and up to 1.42x and 1.38x higher throughput than state-of-the-art high-throughput LLM systems, respectively, on an 8-GPU node. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_02189 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers Zhang, Hongbin Wei, Taosheng Jiang, Jiazhi Yan, Hui Du, Jiangsu Chen, Zhiguang Distributed, Parallel, and Cluster Computing Offline LLM inference seeks to maximize request processing under fixed budgets, making commodity GPU servers a promising choice. However, prior work typically considers offloading and parallelism in isolation, resulting in suboptimal performance. In this paper, we propose PipeMax, a high-throughput LLM inference system that integrates pipeline parallelism with offloading to overcome interconnect and memory constraints on GPU servers. Particularly, pipeline parallelism naturally incurs low communication overhead and keeps only one batch active on each GPU at a time, which enables offloading the KV cache of inactive batches. By coordinating computation with offloading data movement, PipeMax effectively expands GPU memory capacity and sustains large-batch execution. Experiments show that PipeMax achieves up to 2.51x higher throughput than vLLM, and up to 1.42x and 1.38x higher throughput than state-of-the-art high-throughput LLM systems, respectively, on an 8-GPU node. |
| title | PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2605.02189 |