Lumina-Video: Efficient and Flexible Video Generation with Multi-scale Next-DiT
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909489178869760 |
|---|---|
| author | Liu, Dongyang Li, Shicheng Liu, Yutong Li, Zhen Wang, Kai Li, Xinyue Qin, Qi Liu, Yufei Xin, Yi Li, Zhongyu Fu, Bin Si, Chenyang Cao, Yuewen He, Conghui Liu, Ziwei Qiao, Yu Hou, Qibin Li, Hongsheng Gao, Peng |
| author_facet | Liu, Dongyang Li, Shicheng Liu, Yutong Li, Zhen Wang, Kai Li, Xinyue Qin, Qi Liu, Yufei Xin, Yi Li, Zhongyu Fu, Bin Si, Chenyang Cao, Yuewen He, Conghui Liu, Ziwei Qiao, Yu Hou, Qibin Li, Hongsheng Gao, Peng |
| contents | Recent advancements have established Diffusion Transformers (DiTs) as a dominant framework in generative modeling. Building on this success, Lumina-Next achieves exceptional performance in the generation of photorealistic images with Next-DiT. However, its potential for video generation remains largely untapped, with significant challenges in modeling the spatiotemporal complexity inherent to video data. To address this, we introduce Lumina-Video, a framework that leverages the strengths of Next-DiT while introducing tailored solutions for video synthesis. Lumina-Video incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple patchifications to enhance both efficiency and flexibility. By incorporating the motion score as an explicit condition, Lumina-Video also enables direct control of generated videos' dynamic degree. Combined with a progressive training scheme with increasingly higher resolution and FPS, and a multi-source training scheme with mixed natural and synthetic data, Lumina-Video achieves remarkable aesthetic quality and motion smoothness at high training and inference efficiency. We additionally propose Lumina-V2A, a video-to-audio model based on Next-DiT, to create synchronized sounds for generated videos. Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06782 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Lumina-Video: Efficient and Flexible Video Generation with Multi-scale Next-DiT Liu, Dongyang Li, Shicheng Liu, Yutong Li, Zhen Wang, Kai Li, Xinyue Qin, Qi Liu, Yufei Xin, Yi Li, Zhongyu Fu, Bin Si, Chenyang Cao, Yuewen He, Conghui Liu, Ziwei Qiao, Yu Hou, Qibin Li, Hongsheng Gao, Peng Computer Vision and Pattern Recognition Recent advancements have established Diffusion Transformers (DiTs) as a dominant framework in generative modeling. Building on this success, Lumina-Next achieves exceptional performance in the generation of photorealistic images with Next-DiT. However, its potential for video generation remains largely untapped, with significant challenges in modeling the spatiotemporal complexity inherent to video data. To address this, we introduce Lumina-Video, a framework that leverages the strengths of Next-DiT while introducing tailored solutions for video synthesis. Lumina-Video incorporates a Multi-scale Next-DiT architecture, which jointly learns multiple patchifications to enhance both efficiency and flexibility. By incorporating the motion score as an explicit condition, Lumina-Video also enables direct control of generated videos' dynamic degree. Combined with a progressive training scheme with increasingly higher resolution and FPS, and a multi-source training scheme with mixed natural and synthetic data, Lumina-Video achieves remarkable aesthetic quality and motion smoothness at high training and inference efficiency. We additionally propose Lumina-V2A, a video-to-audio model based on Next-DiT, to create synchronized sounds for generated videos. Codes are released at https://www.github.com/Alpha-VLLM/Lumina-Video. |
| title | Lumina-Video: Efficient and Flexible Video Generation with Multi-scale Next-DiT |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.06782 |