Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.08666 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909864024866816 |
|---|---|
| author | Ma, Yuxin Du, Lun Wei, Lanning Chen, Kun Xu, Qian Wang, Kangyu Feng, Guofeng Lu, Guoshan Liu, Lin Qi, Xiaojing Zhang, Xinyuan Tao, Zhen Feng, Haibo Jiang, Ziyun Xu, Ying Huang, Zenan Zhuang, Yihong Xu, Haokai Hu, Jiaqi Lan, Zhenzhong Zhao, Junbo Li, Jianguo Zheng, Da |
| author_facet | Ma, Yuxin Du, Lun Wei, Lanning Chen, Kun Xu, Qian Wang, Kangyu Feng, Guofeng Lu, Guoshan Liu, Lin Qi, Xiaojing Zhang, Xinyuan Tao, Zhen Feng, Haibo Jiang, Ziyun Xu, Ying Huang, Zenan Zhuang, Yihong Xu, Haokai Hu, Jiaqi Lan, Zhenzhong Zhao, Junbo Li, Jianguo Zheng, Da |
| contents | Diffusion-based large language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs, leveraging denoising-based generation to enable inherent parallelism. Even more and more open-sourced dLLM models emerge, yet their widespread adoption remains constrained by the lack of a standardized and efficient inference framework. We present dInfer, an efficient and extensible framework for dLLM inference. dInfer decomposes the inference pipeline into four modular components--model, diffusion iteration manager, decoding strategy, and KV-cache manager--and integrates novel algorithms for each component alongside system-level optimizations. Through this combination of algorithmic innovations and system enhancements, dInfer achieves substantial efficiency gains without compromising output quality on LLaDA-MoE. At batch size 1, it surpasses 1,100 tokens per second on HumanEval and averages over 800 tokens per second across six benchmarks on $8\times$ H800 GPUs. Compared to prior systems, dInfer delivers a $10\times$ speedup over Fast-dLLM while maintaining similar model performance. Even compared to the AR model (with a comparable number of activation parameters and performance) QWen2.5-3B, which is highly optimized with the latest vLLM inference engine, dInfer still delivers a $2$-$3\times$ speedup. The implementation of dInfer is open-sourced at https://github.com/inclusionAI/dInfer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_08666 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | dInfer: An Efficient Inference Framework for Diffusion Language Models Ma, Yuxin Du, Lun Wei, Lanning Chen, Kun Xu, Qian Wang, Kangyu Feng, Guofeng Lu, Guoshan Liu, Lin Qi, Xiaojing Zhang, Xinyuan Tao, Zhen Feng, Haibo Jiang, Ziyun Xu, Ying Huang, Zenan Zhuang, Yihong Xu, Haokai Hu, Jiaqi Lan, Zhenzhong Zhao, Junbo Li, Jianguo Zheng, Da Computation and Language Artificial Intelligence Diffusion-based large language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs, leveraging denoising-based generation to enable inherent parallelism. Even more and more open-sourced dLLM models emerge, yet their widespread adoption remains constrained by the lack of a standardized and efficient inference framework. We present dInfer, an efficient and extensible framework for dLLM inference. dInfer decomposes the inference pipeline into four modular components--model, diffusion iteration manager, decoding strategy, and KV-cache manager--and integrates novel algorithms for each component alongside system-level optimizations. Through this combination of algorithmic innovations and system enhancements, dInfer achieves substantial efficiency gains without compromising output quality on LLaDA-MoE. At batch size 1, it surpasses 1,100 tokens per second on HumanEval and averages over 800 tokens per second across six benchmarks on $8\times$ H800 GPUs. Compared to prior systems, dInfer delivers a $10\times$ speedup over Fast-dLLM while maintaining similar model performance. Even compared to the AR model (with a comparable number of activation parameters and performance) QWen2.5-3B, which is highly optimized with the latest vLLM inference engine, dInfer still delivers a $2$-$3\times$ speedup. The implementation of dInfer is open-sourced at https://github.com/inclusionAI/dInfer. |
| title | dInfer: An Efficient Inference Framework for Diffusion Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.08666 |