DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912928724156416 |
|---|---|
| author | Tan, Xin Chen, Yuetao Jiang, Yimin Chen, Xing Yan, Kun Duan, Nan Zhu, Yibo Jiang, Daxin Xu, Hong |
| author_facet | Tan, Xin Chen, Yuetao Jiang, Yimin Chen, Xing Yan, Kun Duan, Nan Zhu, Yibo Jiang, Daxin Xu, Hong |
| contents | Diffusion Transformers (DiTs) have shown remarkable performance in generating high-quality videos. However, the quadratic complexity of 3D full attention remains a bottleneck in scaling DiT training, especially with high-definition, lengthy videos, where it can consume up to 95% of processing time and demand specialized context parallelism.
This paper introduces DSV to accelerate video DiT training by leveraging the dynamic attention sparsity we empirically observe. DSV uses a two-stage algorithm to capture the dynamic sparsity patterns via low-rank based approximation of the original query and key. It employs custom kernels to efficiently identify critical key-value pairs and compute the sparse attention. To accommodate the new sparsity dimension, DSV adopts a hybrid sparsity-aware context parallelism that re-balances the skewed workload across attention heads and blocks due to sparsity heterogeneity. DSV achieves up to 3.02x higher training throughput, scaling to 128 GPUs and 520k token lengths, without quality loss. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07590 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training Tan, Xin Chen, Yuetao Jiang, Yimin Chen, Xing Yan, Kun Duan, Nan Zhu, Yibo Jiang, Daxin Xu, Hong Distributed, Parallel, and Cluster Computing Computer Vision and Pattern Recognition Diffusion Transformers (DiTs) have shown remarkable performance in generating high-quality videos. However, the quadratic complexity of 3D full attention remains a bottleneck in scaling DiT training, especially with high-definition, lengthy videos, where it can consume up to 95% of processing time and demand specialized context parallelism. This paper introduces DSV to accelerate video DiT training by leveraging the dynamic attention sparsity we empirically observe. DSV uses a two-stage algorithm to capture the dynamic sparsity patterns via low-rank based approximation of the original query and key. It employs custom kernels to efficiently identify critical key-value pairs and compute the sparse attention. To accommodate the new sparsity dimension, DSV adopts a hybrid sparsity-aware context parallelism that re-balances the skewed workload across attention heads and blocks due to sparsity heterogeneity. DSV achieves up to 3.02x higher training throughput, scaling to 128 GPUs and 520k token lengths, without quality loss. |
| title | DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training |
| topic | Distributed, Parallel, and Cluster Computing Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.07590 |