Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.07568 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910004811923456 |
|---|---|
| author | Qian, Yu-Yang Su, Junda Hu, Lanxiang Zhang, Peiyuan Deng, Zhijie Zhao, Peng Zhang, Hao |
| author_facet | Qian, Yu-Yang Su, Junda Hu, Lanxiang Zhang, Peiyuan Deng, Zhijie Zhao, Peng Zhang, Hao |
| contents | Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these benefits in practice is non-trivial, as dLLMs inherently face an accuracy-parallelism trade-off. Despite increasing interest, existing methods typically focus on only one-side of the coin, targeting either efficiency or performance. To address this limitation, we propose d3LLM (Pseudo-Distilled Diffusion Large Language Model), striking a balance between accuracy and parallelism: (i) during training, we introduce pseudo-trajectory distillation to teach the model which tokens can be decoded confidently at early steps, thereby improving parallelism; (ii) during inference, we employ entropy-based multi-block decoding with a KV-cache refresh mechanism to achieve high parallelism while maintaining accuracy. To better evaluate dLLMs, we also introduce AUP (Accuracy Under Parallelism), a new metric that jointly measures accuracy and parallelism. Experiments demonstrate that our d3LLM achieves up to 10$\times$ speedup over vanilla LLaDA/Dream and 5$\times$ speedup over AR models without much accuracy drop. Our code is available at https://github.com/hao-ai-lab/d3LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_07568 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation Qian, Yu-Yang Su, Junda Hu, Lanxiang Zhang, Peiyuan Deng, Zhijie Zhao, Peng Zhang, Hao Machine Learning Artificial Intelligence Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these benefits in practice is non-trivial, as dLLMs inherently face an accuracy-parallelism trade-off. Despite increasing interest, existing methods typically focus on only one-side of the coin, targeting either efficiency or performance. To address this limitation, we propose d3LLM (Pseudo-Distilled Diffusion Large Language Model), striking a balance between accuracy and parallelism: (i) during training, we introduce pseudo-trajectory distillation to teach the model which tokens can be decoded confidently at early steps, thereby improving parallelism; (ii) during inference, we employ entropy-based multi-block decoding with a KV-cache refresh mechanism to achieve high parallelism while maintaining accuracy. To better evaluate dLLMs, we also introduce AUP (Accuracy Under Parallelism), a new metric that jointly measures accuracy and parallelism. Experiments demonstrate that our d3LLM achieves up to 10$\times$ speedup over vanilla LLaDA/Dream and 5$\times$ speedup over AR models without much accuracy drop. Our code is available at https://github.com/hao-ai-lab/d3LLM. |
| title | d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.07568 |