Sequential Diffusion Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909813119647744 |
|---|---|
| author | Liu, Yangzhou Cao, Yue Li, Hao Luo, Gen Chen, Zhe Wang, Weiyun Liang, Xiaobo Qi, Biqing Wu, Lijun Tian, Changyao Zhang, Yanting Li, Yuqiang Lu, Tong Qiao, Yu Dai, Jifeng Wang, Wenhai |
| author_facet | Liu, Yangzhou Cao, Yue Li, Hao Luo, Gen Chen, Zhe Wang, Weiyun Liang, Xiaobo Qi, Biqing Wu, Lijun Tian, Changyao Zhang, Yanting Li, Yuqiang Lu, Tong Qiao, Yu Dai, Jifeng Wang, Wenhai |
| contents | Diffusion language models (DLMs) have strong theoretical efficiency but are limited by fixed-length decoding and incompatibility with key-value (KV) caches. Block diffusion mitigates these issues, yet still enforces a fixed block size and requires expensive training. We introduce Next Sequence Prediction (NSP), which unifies next-token and next-block prediction, enabling the model to adaptively determine the generation length at each step. When the length is fixed to 1, NSP reduces to standard next-token prediction. Building on NSP, we propose Sequential Diffusion Language Model (SDLM), which can retrofit pre-trained autoregressive language models (ALMs) at minimal cost. Specifically, SDLM performs diffusion inference within fixed-size mask blocks, but dynamically decodes consecutive subsequences based on model confidence, thereby preserving KV-cache compatibility and improving robustness to varying uncertainty and semantics across the sequence. Experiments show that SDLM matches or surpasses strong autoregressive baselines using only 3.5M training samples, while achieving 2.1 higher throughput than Qwen-2.5. Notably, the SDLM-32B model delivers even more pronounced efficiency gains, demonstrating the strong scalability potential of our modeling paradigm. Project page and codes: https://github.com/OpenGVLab/SDLM |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24007 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sequential Diffusion Language Models Liu, Yangzhou Cao, Yue Li, Hao Luo, Gen Chen, Zhe Wang, Weiyun Liang, Xiaobo Qi, Biqing Wu, Lijun Tian, Changyao Zhang, Yanting Li, Yuqiang Lu, Tong Qiao, Yu Dai, Jifeng Wang, Wenhai Computation and Language Machine Learning Diffusion language models (DLMs) have strong theoretical efficiency but are limited by fixed-length decoding and incompatibility with key-value (KV) caches. Block diffusion mitigates these issues, yet still enforces a fixed block size and requires expensive training. We introduce Next Sequence Prediction (NSP), which unifies next-token and next-block prediction, enabling the model to adaptively determine the generation length at each step. When the length is fixed to 1, NSP reduces to standard next-token prediction. Building on NSP, we propose Sequential Diffusion Language Model (SDLM), which can retrofit pre-trained autoregressive language models (ALMs) at minimal cost. Specifically, SDLM performs diffusion inference within fixed-size mask blocks, but dynamically decodes consecutive subsequences based on model confidence, thereby preserving KV-cache compatibility and improving robustness to varying uncertainty and semantics across the sequence. Experiments show that SDLM matches or surpasses strong autoregressive baselines using only 3.5M training samples, while achieving 2.1 higher throughput than Qwen-2.5. Notably, the SDLM-32B model delivers even more pronounced efficiency gains, demonstrating the strong scalability potential of our modeling paradigm. Project page and codes: https://github.com/OpenGVLab/SDLM |
| title | Sequential Diffusion Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2509.24007 |