Simple Denoising Diffusion Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908809037873152 |
|---|---|
| author | Zhu, Huaisheng Chen, Zhengyu Zhou, Shijie Xie, Zhihui Yuan, Yige Chen, Shiqi Guo, Zhimeng Xu, Siyuan Zhang, Hangfan Honavar, Vasant Xiao, Teng |
| author_facet | Zhu, Huaisheng Chen, Zhengyu Zhou, Shijie Xie, Zhihui Yuan, Yige Chen, Shiqi Guo, Zhimeng Xu, Siyuan Zhang, Hangfan Honavar, Vasant Xiao, Teng |
| contents | Recent Uniform State Diffusion Models (USDMs), initialized from a uniform prior, offer the promise of fast text generation due to their inherent self-correction ability compared to masked diffusion models. However, they still rely on complex loss formulations with additional computational overhead, which hinders scalability. In this work, we explore a simplified denoising-based loss for USDMs that optimizes only noise-replaced tokens, stabilizing training while matching the performance of prior methods with more complex objectives. In addition, we introduce an efficient regularization term to mitigate corruption toward uniform output distributions, which further improves performance. We demonstrate the effectiveness and efficiency of our simple and improved loss formulations by pretraining models on widely used text datasets for USDMs. More importantly, our conclusions scale to larger models, showing strong potential for large-scale training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22926 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Simple Denoising Diffusion Language Models Zhu, Huaisheng Chen, Zhengyu Zhou, Shijie Xie, Zhihui Yuan, Yige Chen, Shiqi Guo, Zhimeng Xu, Siyuan Zhang, Hangfan Honavar, Vasant Xiao, Teng Machine Learning Recent Uniform State Diffusion Models (USDMs), initialized from a uniform prior, offer the promise of fast text generation due to their inherent self-correction ability compared to masked diffusion models. However, they still rely on complex loss formulations with additional computational overhead, which hinders scalability. In this work, we explore a simplified denoising-based loss for USDMs that optimizes only noise-replaced tokens, stabilizing training while matching the performance of prior methods with more complex objectives. In addition, we introduce an efficient regularization term to mitigate corruption toward uniform output distributions, which further improves performance. We demonstrate the effectiveness and efficiency of our simple and improved loss formulations by pretraining models on widely used text datasets for USDMs. More importantly, our conclusions scale to larger models, showing strong potential for large-scale training. |
| title | Simple Denoising Diffusion Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.22926 |