Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910960154836992 |
|---|---|
| author | Hunter, Rosco Dudziak, Łukasz Abdelfattah, Mohamed S. Mehrotra, Abhinav Bhattacharya, Sourav Wen, Hongkai |
| author_facet | Hunter, Rosco Dudziak, Łukasz Abdelfattah, Mohamed S. Mehrotra, Abhinav Bhattacharya, Sourav Wen, Hongkai |
| contents | Text-to-image diffusion models have demonstrated unprecedented capabilities for flexible and realistic image synthesis. Nevertheless, these models rely on a time-consuming sampling procedure, which has motivated attempts to reduce their latency. When improving efficiency, researchers often use the original diffusion model to train an additional network designed specifically for fast image generation. In contrast, our approach seeks to reduce latency directly, without any retraining, fine-tuning, or knowledge distillation. In particular, we find the repeated calculation of attention maps to be costly yet redundant, and instead suggest reusing them during sampling. Our specific reuse strategies are based on ODE theory, which implies that the later a map is reused, the smaller the distortion in the final image. We empirically compare our reuse strategies with few-step sampling procedures of comparable latency, finding that reuse generates images that are closer to those produced by the original high-latency diffusion model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_01008 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models Hunter, Rosco Dudziak, Łukasz Abdelfattah, Mohamed S. Mehrotra, Abhinav Bhattacharya, Sourav Wen, Hongkai Computer Vision and Pattern Recognition Artificial Intelligence Text-to-image diffusion models have demonstrated unprecedented capabilities for flexible and realistic image synthesis. Nevertheless, these models rely on a time-consuming sampling procedure, which has motivated attempts to reduce their latency. When improving efficiency, researchers often use the original diffusion model to train an additional network designed specifically for fast image generation. In contrast, our approach seeks to reduce latency directly, without any retraining, fine-tuning, or knowledge distillation. In particular, we find the repeated calculation of attention maps to be costly yet redundant, and instead suggest reusing them during sampling. Our specific reuse strategies are based on ODE theory, which implies that the later a map is reused, the smaller the distortion in the final image. We empirically compare our reuse strategies with few-step sampling procedures of comparable latency, finding that reuse generates images that are closer to those produced by the original high-latency diffusion model. |
| title | Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2401.01008 |