Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909536011419648 |
|---|---|
| author | Cui, Jiahao Li, Hui Zhan, Yun Shang, Hanlin Cheng, Kaihui Ma, Yuqi Mu, Shan Zhou, Hang Wang, Jingdong Zhu, Siyu |
| author_facet | Cui, Jiahao Li, Hui Zhan, Yun Shang, Hanlin Cheng, Kaihui Ma, Yuqi Mu, Shan Zhou, Hang Wang, Jingdong Zhu, Siyu |
| contents | Existing methodologies for animating portrait images face significant challenges, particularly in handling non-frontal perspectives, rendering dynamic objects around the portrait, and generating immersive, realistic backgrounds. In this paper, we introduce the first application of a pretrained transformer-based video generative model that demonstrates strong generalization capabilities and generates highly dynamic, realistic videos for portrait animation, effectively addressing these challenges. The adoption of a new video backbone model makes previous U-Net-based methods for identity maintenance, audio conditioning, and video extrapolation inapplicable. To address this limitation, we design an identity reference network consisting of a causal 3D VAE combined with a stacked series of transformer layers, ensuring consistent facial identity across video sequences. Additionally, we investigate various speech audio conditioning and motion frame mechanisms to enable the generation of continuous video driven by speech audio. Our method is validated through experiments on benchmark and newly proposed wild datasets, demonstrating substantial improvements over prior methods in generating realistic portraits characterized by diverse orientations within dynamic and immersive scenes. Further visualizations and the source code are available at: https://fudan-generative-vision.github.io/hallo3/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_00733 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer Cui, Jiahao Li, Hui Zhan, Yun Shang, Hanlin Cheng, Kaihui Ma, Yuqi Mu, Shan Zhou, Hang Wang, Jingdong Zhu, Siyu Computer Vision and Pattern Recognition Graphics Machine Learning Existing methodologies for animating portrait images face significant challenges, particularly in handling non-frontal perspectives, rendering dynamic objects around the portrait, and generating immersive, realistic backgrounds. In this paper, we introduce the first application of a pretrained transformer-based video generative model that demonstrates strong generalization capabilities and generates highly dynamic, realistic videos for portrait animation, effectively addressing these challenges. The adoption of a new video backbone model makes previous U-Net-based methods for identity maintenance, audio conditioning, and video extrapolation inapplicable. To address this limitation, we design an identity reference network consisting of a causal 3D VAE combined with a stacked series of transformer layers, ensuring consistent facial identity across video sequences. Additionally, we investigate various speech audio conditioning and motion frame mechanisms to enable the generation of continuous video driven by speech audio. Our method is validated through experiments on benchmark and newly proposed wild datasets, demonstrating substantial improvements over prior methods in generating realistic portraits characterized by diverse orientations within dynamic and immersive scenes. Further visualizations and the source code are available at: https://fudan-generative-vision.github.io/hallo3/. |
| title | Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer |
| topic | Computer Vision and Pattern Recognition Graphics Machine Learning |
| url | https://arxiv.org/abs/2412.00733 |