GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912750341455872 |
|---|---|
| author | Lin, Yihong Fan, Zhaoxin Wu, Xianjia Xiong, Lingyu Peng, Liang Li, Xiandong Kang, Wenxiong Lei, Songju Xu, Huang |
| author_facet | Lin, Yihong Fan, Zhaoxin Wu, Xianjia Xiong, Lingyu Peng, Liang Li, Xiandong Kang, Wenxiong Lei, Songju Xu, Huang |
| contents | Speech-driven talking head generation is a critical yet challenging task with applications in augmented reality and virtual human modeling. While recent approaches using autoregressive and diffusion-based models have achieved notable progress, they often suffer from modality inconsistencies, particularly misalignment between audio and mesh, leading to reduced motion diversity and lip-sync accuracy. To address this, we propose GLDiTalker, a novel speech-driven 3D facial animation model based on a Graph Latent Diffusion Transformer. GLDiTalker resolves modality misalignment by diffusing signals within a quantized spatiotemporal latent space. It employs a two-stage training pipeline: the Graph-Enhanced Quantized Space Learning Stage ensures lip-sync accuracy, while the Space-Time Powered Latent Diffusion Stage enhances motion diversity. Together, these stages enable GLDiTalker to generate realistic, temporally stable 3D facial animations. Extensive evaluations on standard benchmarks demonstrate that GLDiTalker outperforms existing methods, achieving superior results in both lip-sync accuracy and motion diversity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01826 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer Lin, Yihong Fan, Zhaoxin Wu, Xianjia Xiong, Lingyu Peng, Liang Li, Xiandong Kang, Wenxiong Lei, Songju Xu, Huang Computer Vision and Pattern Recognition Speech-driven talking head generation is a critical yet challenging task with applications in augmented reality and virtual human modeling. While recent approaches using autoregressive and diffusion-based models have achieved notable progress, they often suffer from modality inconsistencies, particularly misalignment between audio and mesh, leading to reduced motion diversity and lip-sync accuracy. To address this, we propose GLDiTalker, a novel speech-driven 3D facial animation model based on a Graph Latent Diffusion Transformer. GLDiTalker resolves modality misalignment by diffusing signals within a quantized spatiotemporal latent space. It employs a two-stage training pipeline: the Graph-Enhanced Quantized Space Learning Stage ensures lip-sync accuracy, while the Space-Time Powered Latent Diffusion Stage enhances motion diversity. Together, these stages enable GLDiTalker to generate realistic, temporally stable 3D facial animations. Extensive evaluations on standard benchmarks demonstrate that GLDiTalker outperforms existing methods, achieving superior results in both lip-sync accuracy and motion diversity. |
| title | GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.01826 |