Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2503.05978 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909531632566272 |
|---|---|
| author | Yi, Hongwei Ye, Tian Shao, Shitong Yang, Xuancheng Zhao, Jiantong Guo, Hanzhong Wang, Terrance Yin, Qingyu Xie, Zeke Zhu, Lei Li, Wei Lingelbach, Michael Zhou, Daquan |
| author_facet | Yi, Hongwei Ye, Tian Shao, Shitong Yang, Xuancheng Zhao, Jiantong Guo, Hanzhong Wang, Terrance Yin, Qingyu Xie, Zeke Zhu, Lei Li, Wei Lingelbach, Michael Zhou, Daquan |
| contents | We present MagicInfinite, a novel diffusion Transformer (DiT) framework that overcomes traditional portrait animation limitations, delivering high-fidelity results across diverse character types-realistic humans, full-body figures, and stylized anime characters. It supports varied facial poses, including back-facing views, and animates single or multiple characters with input masks for precise speaker designation in multi-character scenes. Our approach tackles key challenges with three innovations: (1) 3D full-attention mechanisms with a sliding window denoising strategy, enabling infinite video generation with temporal coherence and visual quality across diverse character styles; (2) a two-stage curriculum learning scheme, integrating audio for lip sync, text for expressive dynamics, and reference images for identity preservation, enabling flexible multi-modal control over long sequences; and (3) region-specific masks with adaptive loss functions to balance global textual control and local audio guidance, supporting speaker-specific animations. Efficiency is enhanced via our innovative unified step and cfg distillation techniques, achieving a 20x inference speed boost over the basemodel: generating a 10 second 540x540p video in 10 seconds or 720x720p in 30 seconds on 8 H100 GPUs, without quality loss. Evaluations on our new benchmark demonstrate MagicInfinite's superiority in audio-lip synchronization, identity preservation, and motion naturalness across diverse scenarios. It is publicly available at https://www.hedra.com/, with examples at https://magicinfinite.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05978 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MagicInfinite: Generating Infinite Talking Videos with Your Words and Voice Yi, Hongwei Ye, Tian Shao, Shitong Yang, Xuancheng Zhao, Jiantong Guo, Hanzhong Wang, Terrance Yin, Qingyu Xie, Zeke Zhu, Lei Li, Wei Lingelbach, Michael Zhou, Daquan Computer Vision and Pattern Recognition We present MagicInfinite, a novel diffusion Transformer (DiT) framework that overcomes traditional portrait animation limitations, delivering high-fidelity results across diverse character types-realistic humans, full-body figures, and stylized anime characters. It supports varied facial poses, including back-facing views, and animates single or multiple characters with input masks for precise speaker designation in multi-character scenes. Our approach tackles key challenges with three innovations: (1) 3D full-attention mechanisms with a sliding window denoising strategy, enabling infinite video generation with temporal coherence and visual quality across diverse character styles; (2) a two-stage curriculum learning scheme, integrating audio for lip sync, text for expressive dynamics, and reference images for identity preservation, enabling flexible multi-modal control over long sequences; and (3) region-specific masks with adaptive loss functions to balance global textual control and local audio guidance, supporting speaker-specific animations. Efficiency is enhanced via our innovative unified step and cfg distillation techniques, achieving a 20x inference speed boost over the basemodel: generating a 10 second 540x540p video in 10 seconds or 720x720p in 30 seconds on 8 H100 GPUs, without quality loss. Evaluations on our new benchmark demonstrate MagicInfinite's superiority in audio-lip synchronization, identity preservation, and motion naturalness across diverse scenarios. It is publicly available at https://www.hedra.com/, with examples at https://magicinfinite.github.io/. |
| title | MagicInfinite: Generating Infinite Talking Videos with Your Words and Voice |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.05978 |