Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910554604437504 |
|---|---|
| author | Tan, Jintao Cheng, Xize Xiong, Lingyu Zhu, Lei Li, Xiandong Wu, Xianjia Gong, Kai Li, Minglei Cai, Yi |
| author_facet | Tan, Jintao Cheng, Xize Xiong, Lingyu Zhu, Lei Li, Xiandong Wu, Xianjia Gong, Kai Li, Minglei Cai, Yi |
| contents | Audio-driven talking head generation is a significant and challenging task applicable to various fields such as virtual avatars, film production, and online conferences. However, the existing GAN-based models emphasize generating well-synchronized lip shapes but overlook the visual quality of generated frames, while diffusion-based models prioritize generating high-quality frames but neglect lip shape matching, resulting in jittery mouth movements. To address the aforementioned problems, we introduce a two-stage diffusion-based model. The first stage involves generating synchronized facial landmarks based on the given speech. In the second stage, these generated landmarks serve as a condition in the denoising process, aiming to optimize mouth jitter issues and generate high-fidelity, well-synchronized, and temporally coherent talking head videos. Extensive experiments demonstrate that our model yields the best performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01732 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation Tan, Jintao Cheng, Xize Xiong, Lingyu Zhu, Lei Li, Xiandong Wu, Xianjia Gong, Kai Li, Minglei Cai, Yi Computer Vision and Pattern Recognition Artificial Intelligence Audio-driven talking head generation is a significant and challenging task applicable to various fields such as virtual avatars, film production, and online conferences. However, the existing GAN-based models emphasize generating well-synchronized lip shapes but overlook the visual quality of generated frames, while diffusion-based models prioritize generating high-quality frames but neglect lip shape matching, resulting in jittery mouth movements. To address the aforementioned problems, we introduce a two-stage diffusion-based model. The first stage involves generating synchronized facial landmarks based on the given speech. In the second stage, these generated landmarks serve as a condition in the denoising process, aiming to optimize mouth jitter issues and generate high-fidelity, well-synchronized, and temporally coherent talking head videos. Extensive experiments demonstrate that our model yields the best performance. |
| title | Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2408.01732 |