Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909595345092608 |
|---|---|
| author | Xi, Haocheng Yang, Shuo Zhao, Yilong Xu, Chenfeng Li, Muyang Li, Xiuyu Lin, Yujun Cai, Han Zhang, Jintao Li, Dacheng Chen, Jianfei Stoica, Ion Keutzer, Kurt Han, Song |
| author_facet | Xi, Haocheng Yang, Shuo Zhao, Yilong Xu, Chenfeng Li, Muyang Li, Xiuyu Lin, Yujun Cai, Han Zhang, Jintao Li, Dacheng Chen, Jianfei Stoica, Ion Keutzer, Kurt Han, Song |
| contents | Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01776 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity Xi, Haocheng Yang, Shuo Zhao, Yilong Xu, Chenfeng Li, Muyang Li, Xiuyu Lin, Yujun Cai, Han Zhang, Jintao Li, Dacheng Chen, Jianfei Stoica, Ion Keutzer, Kurt Han, Song Computer Vision and Pattern Recognition Machine Learning Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen |
| title | Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2502.01776 |