$\textit{Jump Your Steps}$: Optimizing Sampling Schedule of Discrete Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929535582208000 |
|---|---|
| author | Park, Yong-Hyun Lai, Chieh-Hsin Hayakawa, Satoshi Takida, Yuhta Mitsufuji, Yuki |
| author_facet | Park, Yong-Hyun Lai, Chieh-Hsin Hayakawa, Satoshi Takida, Yuhta Mitsufuji, Yuki |
| contents | Diffusion models have seen notable success in continuous domains, leading to the development of discrete diffusion models (DDMs) for discrete variables. Despite recent advances, DDMs face the challenge of slow sampling speeds. While parallel sampling methods like $τ$-leaping accelerate this process, they introduce $\textit{Compounding Decoding Error}$ (CDE), where discrepancies arise between the true distribution and the approximation from parallel token generation, leading to degraded sample quality. In this work, we present $\textit{Jump Your Steps}$ (JYS), a novel approach that optimizes the allocation of discrete sampling timesteps by minimizing CDE without extra computational cost. More precisely, we derive a practical upper bound on CDE and propose an efficient algorithm for searching for the optimal sampling schedule. Extensive experiments across image, music, and text generation show that JYS significantly improves sampling quality, establishing it as a versatile framework for enhancing DDM performance for fast sampling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07761 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | $\textit{Jump Your Steps}$: Optimizing Sampling Schedule of Discrete Diffusion Models Park, Yong-Hyun Lai, Chieh-Hsin Hayakawa, Satoshi Takida, Yuhta Mitsufuji, Yuki Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Diffusion models have seen notable success in continuous domains, leading to the development of discrete diffusion models (DDMs) for discrete variables. Despite recent advances, DDMs face the challenge of slow sampling speeds. While parallel sampling methods like $τ$-leaping accelerate this process, they introduce $\textit{Compounding Decoding Error}$ (CDE), where discrepancies arise between the true distribution and the approximation from parallel token generation, leading to degraded sample quality. In this work, we present $\textit{Jump Your Steps}$ (JYS), a novel approach that optimizes the allocation of discrete sampling timesteps by minimizing CDE without extra computational cost. More precisely, we derive a practical upper bound on CDE and propose an efficient algorithm for searching for the optimal sampling schedule. Extensive experiments across image, music, and text generation show that JYS significantly improves sampling quality, establishing it as a versatile framework for enhancing DDM performance for fast sampling. |
| title | $\textit{Jump Your Steps}$: Optimizing Sampling Schedule of Discrete Diffusion Models |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.07761 |