Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911290994196480 |
|---|---|
| author | Ren, Yinuo Chen, Haoxuan Zhu, Yuchen Guo, Wei Chen, Yongxin Rotskoff, Grant M. Tao, Molei Ying, Lexing |
| author_facet | Ren, Yinuo Chen, Haoxuan Zhu, Yuchen Guo, Wei Chen, Yongxin Rotskoff, Grant M. Tao, Molei Ying, Lexing |
| contents | Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $τ$-leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, $τ$-leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $θ$-Trapezoidal method in KL divergence. Empirical evaluations on GSM8K-level math-reasoning, GPT-2-level text, and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints, with consistent performance gains across models ranging from 200M to 8B. Our code is available at https://github.com/yuchen-zhu-zyc/DiscreteFastSolver. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00234 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms Ren, Yinuo Chen, Haoxuan Zhu, Yuchen Guo, Wei Chen, Yongxin Rotskoff, Grant M. Tao, Molei Ying, Lexing Machine Learning Computer Vision and Pattern Recognition Numerical Analysis Computational Physics Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $τ$-leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, $τ$-leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $θ$-Trapezoidal method in KL divergence. Empirical evaluations on GSM8K-level math-reasoning, GPT-2-level text, and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints, with consistent performance gains across models ranging from 200M to 8B. Our code is available at https://github.com/yuchen-zhu-zyc/DiscreteFastSolver. |
| title | Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms |
| topic | Machine Learning Computer Vision and Pattern Recognition Numerical Analysis Computational Physics |
| url | https://arxiv.org/abs/2502.00234 |