Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908572841934848 |
|---|---|
| author | Zheng, Huangjie Gong, Shansan Zhang, Ruixiang Chen, Tianrong Gu, Jiatao Zhou, Mingyuan Jaitly, Navdeep Zhang, Yizhe |
| author_facet | Zheng, Huangjie Gong, Shansan Zhang, Ruixiang Chen, Tianrong Gu, Jiatao Zhou, Mingyuan Jaitly, Navdeep Zhang, Yizhe |
| contents | Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token. This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps. We introduce Continuously Augmented Discrete Diffusion (CADD), a framework that augments the discrete state space with a paired diffusion in a continuous latent space. This yields graded, gradually corrupted states in which masked tokens are represented by noisy yet informative latent vectors rather than collapsed 'information voids'. At each reverse step, CADD may leverage the continuous latent as a semantic hint to guide discrete denoising. The design is clean and compatible with existing discrete diffusion training. At sampling time, the strength and choice of estimator for the continuous latent vector enables a controlled trade-off between mode-coverage (generating diverse outputs) and mode-seeking (generating contextually precise outputs) behaviors. Empirically, we demonstrate CADD improves generative quality over mask-based diffusion across text generation, image synthesis, and code modeling, with consistent gains on both qualitative and quantitative metrics against strong discrete baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01329 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling Zheng, Huangjie Gong, Shansan Zhang, Ruixiang Chen, Tianrong Gu, Jiatao Zhou, Mingyuan Jaitly, Navdeep Zhang, Yizhe Machine Learning Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token. This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps. We introduce Continuously Augmented Discrete Diffusion (CADD), a framework that augments the discrete state space with a paired diffusion in a continuous latent space. This yields graded, gradually corrupted states in which masked tokens are represented by noisy yet informative latent vectors rather than collapsed 'information voids'. At each reverse step, CADD may leverage the continuous latent as a semantic hint to guide discrete denoising. The design is clean and compatible with existing discrete diffusion training. At sampling time, the strength and choice of estimator for the continuous latent vector enables a controlled trade-off between mode-coverage (generating diverse outputs) and mode-seeking (generating contextually precise outputs) behaviors. Empirically, we demonstrate CADD improves generative quality over mask-based diffusion across text generation, image synthesis, and code modeling, with consistent gains on both qualitative and quantitative metrics against strong discrete baselines. |
| title | Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.01329 |