Plug-and-Play Diffusion Distillation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929386352017408 |
|---|---|
| author | Hsiao, Yi-Ting Khodadadeh, Siavash Duarte, Kevin Lin, Wei-An Qu, Hui Kwon, Mingi Kalarot, Ratheesh |
| author_facet | Hsiao, Yi-Ting Khodadadeh, Siavash Duarte, Kevin Lin, Wei-An Qu, Hui Kwon, Mingi Kalarot, Ratheesh |
| contents | Diffusion models have shown tremendous results in image generation. However, due to the iterative nature of the diffusion process and its reliance on classifier-free guidance, inference times are slow. In this paper, we propose a new distillation approach for guided diffusion models in which an external lightweight guide model is trained while the original text-to-image model remains frozen. We show that our method reduces the inference computation of classifier-free guided latent-space diffusion models by almost half, and only requires 1\% trainable parameters of the base model. Furthermore, once trained, our guide model can be applied to various fine-tuned, domain-specific versions of the base diffusion model without the need for additional training: this "plug-and-play" functionality drastically improves inference computation while maintaining the visual fidelity of generated images. Empirically, we show that our approach is able to produce visually appealing results and achieve a comparable FID score to the teacher with as few as 8 to 16 steps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01954 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Plug-and-Play Diffusion Distillation Hsiao, Yi-Ting Khodadadeh, Siavash Duarte, Kevin Lin, Wei-An Qu, Hui Kwon, Mingi Kalarot, Ratheesh Computer Vision and Pattern Recognition Diffusion models have shown tremendous results in image generation. However, due to the iterative nature of the diffusion process and its reliance on classifier-free guidance, inference times are slow. In this paper, we propose a new distillation approach for guided diffusion models in which an external lightweight guide model is trained while the original text-to-image model remains frozen. We show that our method reduces the inference computation of classifier-free guided latent-space diffusion models by almost half, and only requires 1\% trainable parameters of the base model. Furthermore, once trained, our guide model can be applied to various fine-tuned, domain-specific versions of the base diffusion model without the need for additional training: this "plug-and-play" functionality drastically improves inference computation while maintaining the visual fidelity of generated images. Empirically, we show that our approach is able to produce visually appealing results and achieve a comparable FID score to the teacher with as few as 8 to 16 steps. |
| title | Plug-and-Play Diffusion Distillation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.01954 |