Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908986355220480 |
|---|---|
| author | Hayakawa, Satoshi Takida, Yuhta Imaizumi, Masaaki Wakaki, Hiromi Mitsufuji, Yuki |
| author_facet | Hayakawa, Satoshi Takida, Yuhta Imaizumi, Masaaki Wakaki, Hiromi Mitsufuji, Yuki |
| contents | Masked diffusion models have shown promising performance in generating high-quality samples in a wide range of domains, but accelerating their sampling process remains relatively underexplored. To investigate efficient samplers for masked diffusion, this paper theoretically analyzes the MaskGIT sampler for image modeling, revealing its implicit temperature sampling mechanism. Through this analysis, we introduce the "moment sampler," an asymptotically equivalent but more tractable and interpretable alternative to MaskGIT, which employs a "choose-then-sample" approach by selecting unmasking positions before sampling tokens. In addition, we improve the efficiency of choose-then-sample algorithms through two key innovations: a partial caching technique for transformers that approximates longer sampling trajectories without proportional computational cost, and a hybrid approach formalizing the exploration-exploitation trade-off in adaptive unmasking. Experiments in image and text domains demonstrate our theory as well as the efficiency of our proposed methods, advancing both theoretical understanding and practical implementation of masked diffusion samplers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04525 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion Hayakawa, Satoshi Takida, Yuhta Imaizumi, Masaaki Wakaki, Hiromi Mitsufuji, Yuki Machine Learning Probability Masked diffusion models have shown promising performance in generating high-quality samples in a wide range of domains, but accelerating their sampling process remains relatively underexplored. To investigate efficient samplers for masked diffusion, this paper theoretically analyzes the MaskGIT sampler for image modeling, revealing its implicit temperature sampling mechanism. Through this analysis, we introduce the "moment sampler," an asymptotically equivalent but more tractable and interpretable alternative to MaskGIT, which employs a "choose-then-sample" approach by selecting unmasking positions before sampling tokens. In addition, we improve the efficiency of choose-then-sample algorithms through two key innovations: a partial caching technique for transformers that approximates longer sampling trajectories without proportional computational cost, and a hybrid approach formalizing the exploration-exploitation trade-off in adaptive unmasking. Experiments in image and text domains demonstrate our theory as well as the efficiency of our proposed methods, advancing both theoretical understanding and practical implementation of masked diffusion samplers. |
| title | Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion |
| topic | Machine Learning Probability |
| url | https://arxiv.org/abs/2510.04525 |