RTP-LLM: High-Performance Alibaba LLM Inference Engine
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913169998348288 |
|---|---|
| author | Tan, Boyu Guo, Jiarui Lv, Zongwei Sun, Hanbo Yang, Tong Liu, Kan Shi, Xinfei Hu, Zetao Yu, Yaxin Zhang, Chi Zhang, Jianning Yang, Xi Zhang, Wei Cai, Bo Zhou, Silu Wang, Xiyu He, Na Yu, Yinghao Bao, Wending Huang, Guiyang Yuan, Yuxing Yin, Juncheng Wang, Nan Yang, Lin Zhang, Zechao Chen, Lu Li, Guoding Lan, Tao Qu, Lin |
| author_facet | Tan, Boyu Guo, Jiarui Lv, Zongwei Sun, Hanbo Yang, Tong Liu, Kan Shi, Xinfei Hu, Zetao Yu, Yaxin Zhang, Chi Zhang, Jianning Yang, Xi Zhang, Wei Cai, Bo Zhou, Silu Wang, Xiyu He, Na Yu, Yinghao Bao, Wending Huang, Guiyang Yuan, Yuxing Yin, Juncheng Wang, Nan Yang, Lin Zhang, Zechao Chen, Lu Li, Guoding Lan, Tao Qu, Lin |
| contents | Large Language Models (LLMs) have revolutionized AI applications, but deploying them at scale presents significant challenges. We present RTP-LLM, a high-performance inference engine for industrial-scale LLM deployment, successfully deployed across Alibaba Group serving over 100 million users. RTP-LLM addresses fundamental bottlenecks through integrated design. It optimizes model loading via file-order-driven I/O and parallel I/O-communication overlapping. The Prefill-Decode Disaggregation architecture decouples compute-intensive prefill from memory-bound decode phases, combined with hierarchical multi-tiered KV cache management enabling efficient cache reuse. In addition, RTP-LLM incorporates modular speculative decoding supporting multiple algorithms, adaptive KV cache quantization, and decoupled multimodal processing, with support for multi-level parallelism.
Comprehensive evaluations across diverse model architectures (8B-235B parameters) have been conducted, where both controlled benchmarks and real production workloads are used. The results demonstrate RTP-LLM's superior performance against vLLM and SGLang: 4.7x-6.3x model loading speedup, 35-37% TTFT P95 latency reduction with 215% cache reuse improvement in production traffic scheduling, 1.12x-2.48x and 1.86x-2.52x throughput improvements in speculative decoding and multimodal inference, respectively, and 35-40% batch latency reduction with 1.9x-3.0x TTFT improvement in quantized inference. RTP-LLM's production-proven architecture and open-source availability make it a comprehensive solution for industrial LLM deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_29639 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RTP-LLM: High-Performance Alibaba LLM Inference Engine Tan, Boyu Guo, Jiarui Lv, Zongwei Sun, Hanbo Yang, Tong Liu, Kan Shi, Xinfei Hu, Zetao Yu, Yaxin Zhang, Chi Zhang, Jianning Yang, Xi Zhang, Wei Cai, Bo Zhou, Silu Wang, Xiyu He, Na Yu, Yinghao Bao, Wending Huang, Guiyang Yuan, Yuxing Yin, Juncheng Wang, Nan Yang, Lin Zhang, Zechao Chen, Lu Li, Guoding Lan, Tao Qu, Lin Operating Systems Large Language Models (LLMs) have revolutionized AI applications, but deploying them at scale presents significant challenges. We present RTP-LLM, a high-performance inference engine for industrial-scale LLM deployment, successfully deployed across Alibaba Group serving over 100 million users. RTP-LLM addresses fundamental bottlenecks through integrated design. It optimizes model loading via file-order-driven I/O and parallel I/O-communication overlapping. The Prefill-Decode Disaggregation architecture decouples compute-intensive prefill from memory-bound decode phases, combined with hierarchical multi-tiered KV cache management enabling efficient cache reuse. In addition, RTP-LLM incorporates modular speculative decoding supporting multiple algorithms, adaptive KV cache quantization, and decoupled multimodal processing, with support for multi-level parallelism. Comprehensive evaluations across diverse model architectures (8B-235B parameters) have been conducted, where both controlled benchmarks and real production workloads are used. The results demonstrate RTP-LLM's superior performance against vLLM and SGLang: 4.7x-6.3x model loading speedup, 35-37% TTFT P95 latency reduction with 215% cache reuse improvement in production traffic scheduling, 1.12x-2.48x and 1.86x-2.52x throughput improvements in speculative decoding and multimodal inference, respectively, and 35-40% batch latency reduction with 1.9x-3.0x TTFT improvement in quantized inference. RTP-LLM's production-proven architecture and open-source availability make it a comprehensive solution for industrial LLM deployment. |
| title | RTP-LLM: High-Performance Alibaba LLM Inference Engine |
| topic | Operating Systems |
| url | https://arxiv.org/abs/2605.29639 |