Understanding and Accelerating the Training of Masked Diffusion Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909039369125888 |
|---|---|
| author | Hong, Chunsan Lee, Sanghyun Lai, Chieh-Hsin Hayakawa, Satoshi Takida, Yuhta Mitsufuji, Yuki Kim, Seungryong Ye, Jong Chul |
| author_facet | Hong, Chunsan Lee, Sanghyun Lai, Chieh-Hsin Hayakawa, Satoshi Takida, Yuhta Mitsufuji, Yuki Kim, Seungryong Ye, Jong Chul |
| contents | Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling. However, MDMs are known to learn substantially more slowly than ARMs, which may become problematic when scaling MDMs to larger models. Therefore, we ask the following question: how can we accelerate standard MDM training while maintaining its final performance? To this end, we first provide a detailed analysis of why MDM training is slow. We find that the main factor is the locality bias of language: the predictive information for a token is concentrated in nearby positions. We further investigate how this bias slows learning and suggest a simple yet effective remedy: bell-shaped time sampling as a training strategy. Notably, MDMs trained with our training recipe reach the same validation negative log-likelihood (NLL) up to $\sim4\times$ faster than standard training on One Billion Word Benchmark (LM1B). We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_13026 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Understanding and Accelerating the Training of Masked Diffusion Language Models Hong, Chunsan Lee, Sanghyun Lai, Chieh-Hsin Hayakawa, Satoshi Takida, Yuhta Mitsufuji, Yuki Kim, Seungryong Ye, Jong Chul Machine Learning Artificial Intelligence Computation and Language Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling. However, MDMs are known to learn substantially more slowly than ARMs, which may become problematic when scaling MDMs to larger models. Therefore, we ask the following question: how can we accelerate standard MDM training while maintaining its final performance? To this end, we first provide a detailed analysis of why MDM training is slow. We find that the main factor is the locality bias of language: the predictive information for a token is concentrated in nearby positions. We further investigate how this bias slows learning and suggest a simple yet effective remedy: bell-shaped time sampling as a training strategy. Notably, MDMs trained with our training recipe reach the same validation negative log-likelihood (NLL) up to $\sim4\times$ faster than standard training on One Billion Word Benchmark (LM1B). We also show faster improvements in generative perplexity, zero-shot perplexity, and downstream task performance on various benchmarks. |
| title | Understanding and Accelerating the Training of Masked Diffusion Language Models |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2605.13026 |