xLLM Technical Report
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910038526787584 |
|---|---|
| author | Liu, Tongxuan Peng, Tao Yang, Peijun Zhao, Xiaoyang Lu, Xiusheng Huang, Weizhe Liu, Zirui Chen, Xiaoyu Liang, Zhiwei Xiong, Jun Jin, Donghe Zhang, Minchao Guo, Jinrong Deng, Yingxu Zhang, Xu Dong, Xianzhe Wang, Siqi Wu, Siyu Wu, Yu Tang, Zihan Zeng, Yuting Wang, Yanshu Liu, Jinguang Kang, Meng Li, Menxin Wang, Yunlong Liu, Yiming Ma, Xiaolong Wang, Yifan Zhang, Yichen Yin, Jinrun Zheng, Keyang Yin, Jiawei Zhang, Jun Wang, Ziyue Lin, Xiaobo Liu, Liangyu Lan, Liwei Liu, Yang Peng, Chunhua Liu, Han Ren, Songcheng Wang, Xuezhu Shen, Yunheng Wang, Yi Liu, Guyue Hu, Yitao Chen, Hui Yang, Tong Yang, Hailong Li, Jing Ding, Guiguang Zhang, Ke |
| author_facet | Liu, Tongxuan Peng, Tao Yang, Peijun Zhao, Xiaoyang Lu, Xiusheng Huang, Weizhe Liu, Zirui Chen, Xiaoyu Liang, Zhiwei Xiong, Jun Jin, Donghe Zhang, Minchao Guo, Jinrong Deng, Yingxu Zhang, Xu Dong, Xianzhe Wang, Siqi Wu, Siyu Wu, Yu Tang, Zihan Zeng, Yuting Wang, Yanshu Liu, Jinguang Kang, Meng Li, Menxin Wang, Yunlong Liu, Yiming Ma, Xiaolong Wang, Yifan Zhang, Yichen Yin, Jinrun Zheng, Keyang Yin, Jiawei Zhang, Jun Wang, Ziyue Lin, Xiaobo Liu, Liangyu Lan, Liwei Liu, Yang Peng, Chunhua Liu, Han Ren, Songcheng Wang, Xuezhu Shen, Yunheng Wang, Yi Liu, Guyue Hu, Yitao Chen, Hui Yang, Tong Yang, Hailong Li, Jing Ding, Guiguang Zhang, Ke |
| contents | We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework designed for high-performance, large-scale enterprise-grade serving, with deep optimizations for diverse AI accelerators. To address these challenges, xLLM builds a novel decoupled service-engine architecture. At the service layer, xLLM-Service features an intelligent scheduling module that efficiently processes multimodal requests and co-locates online and offline tasks through unified elastic scheduling to maximize cluster utilization. This module also relies on a workload-adaptive dynamic Prefill-Decode (PD) disaggregation policy and a novel Encode-Prefill-Decode (EPD) disaggregation policy designed for multimodal inputs. Furthermore, it incorporates a distributed architecture to provide global KV Cache management and robust fault-tolerant capabilities for high availability. At the engine layer, xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources. This is achieved through comprehensive multi-layer execution pipeline optimizations, an adaptive graph mode and an xTensor memory management. xLLM-Engine also further integrates algorithmic enhancements such as optimized speculative decoding and dynamic EPLB, collectively serving to substantially boost throughput and inference efficiency. Extensive evaluations demonstrate that xLLM delivers significantly superior performance and resource efficiency. Under identical TPOT constraints, xLLM achieves throughput up to 1.7x that of MindIE and 2.2x that of vLLM-Ascend with Qwen-series models, while maintaining an average throughput of 1.7x that of MindIE with Deepseek-series models. xLLM framework is publicly available at https://github.com/jd-opensource/xllm and https://github.com/jd-opensource/xllm-service. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_14686 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | xLLM Technical Report Liu, Tongxuan Peng, Tao Yang, Peijun Zhao, Xiaoyang Lu, Xiusheng Huang, Weizhe Liu, Zirui Chen, Xiaoyu Liang, Zhiwei Xiong, Jun Jin, Donghe Zhang, Minchao Guo, Jinrong Deng, Yingxu Zhang, Xu Dong, Xianzhe Wang, Siqi Wu, Siyu Wu, Yu Tang, Zihan Zeng, Yuting Wang, Yanshu Liu, Jinguang Kang, Meng Li, Menxin Wang, Yunlong Liu, Yiming Ma, Xiaolong Wang, Yifan Zhang, Yichen Yin, Jinrun Zheng, Keyang Yin, Jiawei Zhang, Jun Wang, Ziyue Lin, Xiaobo Liu, Liangyu Lan, Liwei Liu, Yang Peng, Chunhua Liu, Han Ren, Songcheng Wang, Xuezhu Shen, Yunheng Wang, Yi Liu, Guyue Hu, Yitao Chen, Hui Yang, Tong Yang, Hailong Li, Jing Ding, Guiguang Zhang, Ke Distributed, Parallel, and Cluster Computing Artificial Intelligence We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework designed for high-performance, large-scale enterprise-grade serving, with deep optimizations for diverse AI accelerators. To address these challenges, xLLM builds a novel decoupled service-engine architecture. At the service layer, xLLM-Service features an intelligent scheduling module that efficiently processes multimodal requests and co-locates online and offline tasks through unified elastic scheduling to maximize cluster utilization. This module also relies on a workload-adaptive dynamic Prefill-Decode (PD) disaggregation policy and a novel Encode-Prefill-Decode (EPD) disaggregation policy designed for multimodal inputs. Furthermore, it incorporates a distributed architecture to provide global KV Cache management and robust fault-tolerant capabilities for high availability. At the engine layer, xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources. This is achieved through comprehensive multi-layer execution pipeline optimizations, an adaptive graph mode and an xTensor memory management. xLLM-Engine also further integrates algorithmic enhancements such as optimized speculative decoding and dynamic EPLB, collectively serving to substantially boost throughput and inference efficiency. Extensive evaluations demonstrate that xLLM delivers significantly superior performance and resource efficiency. Under identical TPOT constraints, xLLM achieves throughput up to 1.7x that of MindIE and 2.2x that of vLLM-Ascend with Qwen-series models, while maintaining an average throughput of 1.7x that of MindIE with Deepseek-series models. xLLM framework is publicly available at https://github.com/jd-opensource/xllm and https://github.com/jd-opensource/xllm-service. |
| title | xLLM Technical Report |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2510.14686 |