Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911722387800064 |
|---|---|
| author | Shabalin, Alexander Elistratov, Simon Meshchaninov, Viacheslav Sadrtdinov, Ildus Vetrov, Dmitry |
| author_facet | Shabalin, Alexander Elistratov, Simon Meshchaninov, Viacheslav Sadrtdinov, Ildus Vetrov, Dmitry |
| contents | Diffusion models have become a standard approach for generative modeling in continuous domains, yet their application to discrete data remains challenging. We investigate why Gaussian diffusion models with the DDPM solver struggle to sample from discrete distributions that are represented as a mixture of delta-distributions in the continuous space. Using a toy Random Hierarchy Model, we identify a critical sampling interval in which the density of noisified data becomes multimodal. In this regime, DDPM occasionally enters low-density regions between modes producing out-of-distribution inputs for the model and degrading sample quality. We show that existing heuristics, including self-conditioning and a solver we term q-sampling, help alleviate this issue. Furthermore, we demonstrate that combining self-conditioning with switching from DDPM to q-sampling within the critical interval improves generation quality on real data. We validate these findings across conditional and unconditional tasks in multiple domains, including text, programming code, and proteins. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_02028 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It? Shabalin, Alexander Elistratov, Simon Meshchaninov, Viacheslav Sadrtdinov, Ildus Vetrov, Dmitry Computation and Language Diffusion models have become a standard approach for generative modeling in continuous domains, yet their application to discrete data remains challenging. We investigate why Gaussian diffusion models with the DDPM solver struggle to sample from discrete distributions that are represented as a mixture of delta-distributions in the continuous space. Using a toy Random Hierarchy Model, we identify a critical sampling interval in which the density of noisified data becomes multimodal. In this regime, DDPM occasionally enters low-density regions between modes producing out-of-distribution inputs for the model and degrading sample quality. We show that existing heuristics, including self-conditioning and a solver we term q-sampling, help alleviate this issue. Furthermore, we demonstrate that combining self-conditioning with switching from DDPM to q-sampling within the critical interval improves generation quality on real data. We validate these findings across conditional and unconditional tasks in multiple domains, including text, programming code, and proteins. |
| title | Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.02028 |