FullDiT: Multi-Task Video Generative Foundation Model with Full Attention
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913757744070656 |
|---|---|
| author | Ju, Xuan Ye, Weicai Liu, Quande Wang, Qiulin Wang, Xintao Wan, Pengfei Zhang, Di Gai, Kun Xu, Qiang |
| author_facet | Ju, Xuan Ye, Weicai Liu, Quande Wang, Qiulin Wang, Xintao Wan, Pengfei Zhang, Di Gai, Kun Xu, Qiang |
| contents | Current video generative foundation models primarily focus on text-to-video tasks, providing limited control for fine-grained video content creation. Although adapter-based approaches (e.g., ControlNet) enable additional controls with minimal fine-tuning, they encounter challenges when integrating multiple conditions, including: branch conflicts between independently trained adapters, parameter redundancy leading to increased computational cost, and suboptimal performance compared to full fine-tuning. To address these challenges, we introduce FullDiT, a unified foundation model for video generation that seamlessly integrates multiple conditions via unified full-attention mechanisms. By fusing multi-task conditions into a unified sequence representation and leveraging the long-context learning ability of full self-attention to capture condition dynamics, FullDiT reduces parameter overhead, avoids conditions conflict, and shows scalability and emergent ability. We further introduce FullBench for multi-task video generation evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art results, highlighting the efficacy of full-attention in complex multi-task video generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_19907 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FullDiT: Multi-Task Video Generative Foundation Model with Full Attention Ju, Xuan Ye, Weicai Liu, Quande Wang, Qiulin Wang, Xintao Wan, Pengfei Zhang, Di Gai, Kun Xu, Qiang Computer Vision and Pattern Recognition Current video generative foundation models primarily focus on text-to-video tasks, providing limited control for fine-grained video content creation. Although adapter-based approaches (e.g., ControlNet) enable additional controls with minimal fine-tuning, they encounter challenges when integrating multiple conditions, including: branch conflicts between independently trained adapters, parameter redundancy leading to increased computational cost, and suboptimal performance compared to full fine-tuning. To address these challenges, we introduce FullDiT, a unified foundation model for video generation that seamlessly integrates multiple conditions via unified full-attention mechanisms. By fusing multi-task conditions into a unified sequence representation and leveraging the long-context learning ability of full self-attention to capture condition dynamics, FullDiT reduces parameter overhead, avoids conditions conflict, and shows scalability and emergent ability. We further introduce FullBench for multi-task video generation evaluation. Experiments demonstrate that FullDiT achieves state-of-the-art results, highlighting the efficacy of full-attention in complex multi-task video generation. |
| title | FullDiT: Multi-Task Video Generative Foundation Model with Full Attention |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.19907 |