DiTFastAttn: Attention Compression for Diffusion Transformer Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917807659155456 |
|---|---|
| author | Yuan, Zhihang Zhang, Hanling Lu, Pu Ning, Xuefei Zhang, Linfeng Zhao, Tianchen Yan, Shengen Dai, Guohao Wang, Yu |
| author_facet | Yuan, Zhihang Zhang, Hanling Lu, Pu Ning, Xuefei Zhang, Linfeng Zhao, Tianchen Yan, Shengen Dai, Guohao Wang, Yu |
| contents | Diffusion Transformers (DiT) excel at image and video generation but face computational challenges due to the quadratic complexity of self-attention operators. We propose DiTFastAttn, a post-training compression method to alleviate the computational bottleneck of DiT. We identify three key redundancies in the attention computation during DiT inference: (1) spatial redundancy, where many attention heads focus on local information; (2) temporal redundancy, with high similarity between the attention outputs of neighboring steps; (3) conditional redundancy, where conditional and unconditional inferences exhibit significant similarity. We propose three techniques to reduce these redundancies: (1) Window Attention with Residual Sharing to reduce spatial redundancy; (2) Attention Sharing across Timesteps to exploit the similarity between steps; (3) Attention Sharing across CFG to skip redundant computations during conditional generation. We apply DiTFastAttn to DiT, PixArt-Sigma for image generation tasks, and OpenSora for video generation tasks. Our results show that for image generation, our method reduces up to 76% of the attention FLOPs and achieves up to 1.8x end-to-end speedup at high-resolution (2k x 2k) generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08552 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DiTFastAttn: Attention Compression for Diffusion Transformer Models Yuan, Zhihang Zhang, Hanling Lu, Pu Ning, Xuefei Zhang, Linfeng Zhao, Tianchen Yan, Shengen Dai, Guohao Wang, Yu Computer Vision and Pattern Recognition Diffusion Transformers (DiT) excel at image and video generation but face computational challenges due to the quadratic complexity of self-attention operators. We propose DiTFastAttn, a post-training compression method to alleviate the computational bottleneck of DiT. We identify three key redundancies in the attention computation during DiT inference: (1) spatial redundancy, where many attention heads focus on local information; (2) temporal redundancy, with high similarity between the attention outputs of neighboring steps; (3) conditional redundancy, where conditional and unconditional inferences exhibit significant similarity. We propose three techniques to reduce these redundancies: (1) Window Attention with Residual Sharing to reduce spatial redundancy; (2) Attention Sharing across Timesteps to exploit the similarity between steps; (3) Attention Sharing across CFG to skip redundant computations during conditional generation. We apply DiTFastAttn to DiT, PixArt-Sigma for image generation tasks, and OpenSora for video generation tasks. Our results show that for image generation, our method reduces up to 76% of the attention FLOPs and achieves up to 1.8x end-to-end speedup at high-resolution (2k x 2k) generation. |
| title | DiTFastAttn: Attention Compression for Diffusion Transformer Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.08552 |