ARTalk: Speech-Driven 3D Head Animation via Autoregressive Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911137731182592 |
|---|---|
| author | Chu, Xuangeng Goswami, Nabarun Cui, Ziteng Wang, Hanqin Harada, Tatsuya |
| author_facet | Chu, Xuangeng Goswami, Nabarun Cui, Ziteng Wang, Hanqin Harada, Tatsuya |
| contents | Speech-driven 3D facial animation aims to generate realistic lip movements and facial expressions for 3D head models from arbitrary audio clips. Although existing diffusion-based methods are capable of producing natural motions, their slow generation speed limits their application potential. In this paper, we introduce a novel autoregressive model that achieves real-time generation of highly synchronized lip movements and realistic head poses and eye blinks by learning a mapping from speech to a multi-scale motion codebook. Furthermore, our model can adapt to unseen speaking styles, enabling the creation of 3D talking avatars with unique personal styles beyond the identities seen during training. Extensive evaluations and user studies demonstrate that our method outperforms existing approaches in lip synchronization accuracy and perceived quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20323 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ARTalk: Speech-Driven 3D Head Animation via Autoregressive Model Chu, Xuangeng Goswami, Nabarun Cui, Ziteng Wang, Hanqin Harada, Tatsuya Computer Vision and Pattern Recognition Speech-driven 3D facial animation aims to generate realistic lip movements and facial expressions for 3D head models from arbitrary audio clips. Although existing diffusion-based methods are capable of producing natural motions, their slow generation speed limits their application potential. In this paper, we introduce a novel autoregressive model that achieves real-time generation of highly synchronized lip movements and realistic head poses and eye blinks by learning a mapping from speech to a multi-scale motion codebook. Furthermore, our model can adapt to unseen speaking styles, enabling the creation of 3D talking avatars with unique personal styles beyond the identities seen during training. Extensive evaluations and user studies demonstrate that our method outperforms existing approaches in lip synchronization accuracy and perceived quality. |
| title | ARTalk: Speech-Driven 3D Head Animation via Autoregressive Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.20323 |