MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911224371871744 |
|---|---|
| author | Jia, Weinan Lu, Yuning Huang, Mengqi Wang, Hualiang Huang, Binyuan Chen, Nan Liu, Mu Jiang, Jidong Mao, Zhendong |
| author_facet | Jia, Weinan Lu, Yuning Huang, Mengqi Wang, Hualiang Huang, Binyuan Chen, Nan Liu, Mu Jiang, Jidong Mao, Zhendong |
| contents | Long video generation with Diffusion Transformers (DiTs) is bottlenecked by the quadratic scaling of full attention with sequence length. Since attention is highly redundant, outputs are dominated by a small subset of query-key pairs. Existing sparse methods rely on blockwise coarse estimation, whose accuracy-efficiency trade-offs are constrained by block size. This paper introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention that uses a lightweight, learnable token router to precisely match tokens without blockwise estimation. Through semantic-aware routing, MoGA enables effective long-range interactions. As a kernel-free method, MoGA integrates seamlessly with modern attention stacks, including FlashAttention and sequence parallelism. Building on MoGA, we develop an efficient long video generation model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps, with a context length of approximately 580k. Comprehensive experiments on various video generation tasks validate the effectiveness of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_18692 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation Jia, Weinan Lu, Yuning Huang, Mengqi Wang, Hualiang Huang, Binyuan Chen, Nan Liu, Mu Jiang, Jidong Mao, Zhendong Computer Vision and Pattern Recognition Long video generation with Diffusion Transformers (DiTs) is bottlenecked by the quadratic scaling of full attention with sequence length. Since attention is highly redundant, outputs are dominated by a small subset of query-key pairs. Existing sparse methods rely on blockwise coarse estimation, whose accuracy-efficiency trade-offs are constrained by block size. This paper introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention that uses a lightweight, learnable token router to precisely match tokens without blockwise estimation. Through semantic-aware routing, MoGA enables effective long-range interactions. As a kernel-free method, MoGA integrates seamlessly with modern attention stacks, including FlashAttention and sequence parallelism. Building on MoGA, we develop an efficient long video generation model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps, with a context length of approximately 580k. Comprehensive experiments on various video generation tasks validate the effectiveness of our approach. |
| title | MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.18692 |