AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911814279757824 |
|---|---|
| author | Wei, Huawei Yang, Zejun Wang, Zhisheng |
| author_facet | Wei, Huawei Yang, Zejun Wang, Zhisheng |
| contents | In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority of AniPortrait in terms of facial naturalness, pose diversity, and visual quality, thereby offering an enhanced perceptual experience. Moreover, our methodology exhibits considerable potential in terms of flexibility and controllability, which can be effectively applied in areas such as facial motion editing or face reenactment. We release code and model weights at https://github.com/scutzzj/AniPortrait |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_17694 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation Wei, Huawei Yang, Zejun Wang, Zhisheng Computer Vision and Pattern Recognition Graphics Image and Video Processing In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority of AniPortrait in terms of facial naturalness, pose diversity, and visual quality, thereby offering an enhanced perceptual experience. Moreover, our methodology exhibits considerable potential in terms of flexibility and controllability, which can be effectively applied in areas such as facial motion editing or face reenactment. We release code and model weights at https://github.com/scutzzj/AniPortrait |
| title | AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation |
| topic | Computer Vision and Pattern Recognition Graphics Image and Video Processing |
| url | https://arxiv.org/abs/2403.17694 |