Simple and Effective Masked Diffusion Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916475209515008 |
|---|---|
| author | Sahoo, Subham Sekhar Arriola, Marianne Schiff, Yair Gokaslan, Aaron Marroquin, Edgar Chiu, Justin T Rush, Alexander Kuleshov, Volodymyr |
| author_facet | Sahoo, Subham Sekhar Arriola, Marianne Schiff, Yair Gokaslan, Aaron Marroquin, Edgar Chiu, Justin T Rush, Alexander Kuleshov, Volodymyr |
| contents | While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07524 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Simple and Effective Masked Diffusion Language Models Sahoo, Subham Sekhar Arriola, Marianne Schiff, Yair Gokaslan, Aaron Marroquin, Edgar Chiu, Justin T Rush, Alexander Kuleshov, Volodymyr Computation and Language Artificial Intelligence Machine Learning While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm |
| title | Simple and Effective Masked Diffusion Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2406.07524 |