Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.08128 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917594350485504 |
|---|---|
| author | Habibian, Amirhossein Ghodrati, Amir Fathima, Noor Sautiere, Guillaume Garrepalli, Risheek Porikli, Fatih Petersen, Jens |
| author_facet | Habibian, Amirhossein Ghodrati, Amir Fathima, Noor Sautiere, Guillaume Garrepalli, Risheek Porikli, Fatih Petersen, Jens |
| contents | This work aims to improve the efficiency of text-to-image diffusion models. While diffusion models use computationally expensive UNet-based denoising operations in every generation step, we identify that not all operations are equally relevant for the final output quality. In particular, we observe that UNet layers operating on high-res feature maps are relatively sensitive to small perturbations. In contrast, low-res feature maps influence the semantic layout of the final image and can often be perturbed with no noticeable change in the output. Based on this observation, we propose Clockwork Diffusion, a method that periodically reuses computation from preceding denoising steps to approximate low-res feature maps at one or more subsequent steps. For multiple baselines, and for both text-to-image generation and image editing, we demonstrate that Clockwork leads to comparable or improved perceptual scores with drastically reduced computational complexity. As an example, for Stable Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and CLIP change. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_08128 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Clockwork Diffusion: Efficient Generation With Model-Step Distillation Habibian, Amirhossein Ghodrati, Amir Fathima, Noor Sautiere, Guillaume Garrepalli, Risheek Porikli, Fatih Petersen, Jens Computer Vision and Pattern Recognition This work aims to improve the efficiency of text-to-image diffusion models. While diffusion models use computationally expensive UNet-based denoising operations in every generation step, we identify that not all operations are equally relevant for the final output quality. In particular, we observe that UNet layers operating on high-res feature maps are relatively sensitive to small perturbations. In contrast, low-res feature maps influence the semantic layout of the final image and can often be perturbed with no noticeable change in the output. Based on this observation, we propose Clockwork Diffusion, a method that periodically reuses computation from preceding denoising steps to approximate low-res feature maps at one or more subsequent steps. For multiple baselines, and for both text-to-image generation and image editing, we demonstrate that Clockwork leads to comparable or improved perceptual scores with drastically reduced computational complexity. As an example, for Stable Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and CLIP change. |
| title | Clockwork Diffusion: Efficient Generation With Model-Step Distillation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.08128 |