Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929542066601984 |
|---|---|
| author | Li, Senmao Hu, Taihang van de Weijer, Joost Khan, Fahad Shahbaz Liu, Tao Li, Linxuan Yang, Shiqi Wang, Yaxing Cheng, Ming-Ming Yang, Jian |
| author_facet | Li, Senmao Hu, Taihang van de Weijer, Joost Khan, Fahad Shahbaz Liu, Tao Li, Linxuan Yang, Shiqi Wang, Yaxing Cheng, Ming-Ming Yang, Jian |
| contents | One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally, we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41$\%$ and 24$\%$ respectively, and DiT model sampling by 34$\%$, while maintaining high-quality generation performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_09608 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference Li, Senmao Hu, Taihang van de Weijer, Joost Khan, Fahad Shahbaz Liu, Tao Li, Linxuan Yang, Shiqi Wang, Yaxing Cheng, Ming-Ming Yang, Jian Computer Vision and Pattern Recognition One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally, we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41$\%$ and 24$\%$ respectively, and DiT model sampling by 34$\%$, while maintaining high-quality generation performance. |
| title | Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.09608 |