Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.15892 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914274698330112 |
|---|---|
| author | Fan, Chenghao Heng, Wen Li, Bo Liu, Sichen Song, Yuxuan Su, Jing Qu, Xiaoye Shen, Kai Wei, Wei |
| author_facet | Fan, Chenghao Heng, Wen Li, Bo Liu, Sichen Song, Yuxuan Su, Jing Qu, Xiaoye Shen, Kai Wei, Wei |
| contents | Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_15892 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model Fan, Chenghao Heng, Wen Li, Bo Liu, Sichen Song, Yuxuan Su, Jing Qu, Xiaoye Shen, Kai Wei, Wei Computation and Language Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages. |
| title | Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.15892 |