Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909224539258880 |
|---|---|
| author | Xu, Mingwang Li, Hui Su, Qingkun Shang, Hanlin Zhang, Liwei Liu, Ce Wang, Jingdong Yao, Yao Zhu, Siyu |
| author_facet | Xu, Mingwang Li, Hui Su, Qingkun Shang, Hanlin Zhang, Liwei Liu, Ce Wang, Jingdong Yao, Yao Zhu, Siyu |
| contents | The field of portrait image animation, driven by speech audio input, has experienced significant advancements in the generation of realistic and dynamic portraits. This research delves into the complexities of synchronizing facial movements and creating visually appealing, temporally consistent animations within the framework of diffusion-based methodologies. Moving away from traditional paradigms that rely on parametric models for intermediate facial representations, our innovative approach embraces the end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module to enhance the precision of alignment between audio inputs and visual outputs, encompassing lip, expression, and pose motion. Our proposed network architecture seamlessly integrates diffusion-based generative models, a UNet-based denoiser, temporal alignment techniques, and a reference network. The proposed hierarchical audio-driven visual synthesis offers adaptive control over expression and pose diversity, enabling more effective personalization tailored to different identities. Through a comprehensive evaluation that incorporates both qualitative and quantitative analyses, our approach demonstrates obvious enhancements in image and video quality, lip synchronization precision, and motion diversity. Further visualization and access to the source code can be found at: https://fudan-generative-vision.github.io/hallo. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08801 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation Xu, Mingwang Li, Hui Su, Qingkun Shang, Hanlin Zhang, Liwei Liu, Ce Wang, Jingdong Yao, Yao Zhu, Siyu Computer Vision and Pattern Recognition The field of portrait image animation, driven by speech audio input, has experienced significant advancements in the generation of realistic and dynamic portraits. This research delves into the complexities of synchronizing facial movements and creating visually appealing, temporally consistent animations within the framework of diffusion-based methodologies. Moving away from traditional paradigms that rely on parametric models for intermediate facial representations, our innovative approach embraces the end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module to enhance the precision of alignment between audio inputs and visual outputs, encompassing lip, expression, and pose motion. Our proposed network architecture seamlessly integrates diffusion-based generative models, a UNet-based denoiser, temporal alignment techniques, and a reference network. The proposed hierarchical audio-driven visual synthesis offers adaptive control over expression and pose diversity, enabling more effective personalization tailored to different identities. Through a comprehensive evaluation that incorporates both qualitative and quantitative analyses, our approach demonstrates obvious enhancements in image and video quality, lip synchronization precision, and motion diversity. Further visualization and access to the source code can be found at: https://fudan-generative-vision.github.io/hallo. |
| title | Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.08801 |