Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917935185920000 |
|---|---|
| author | Chopin, Baptiste Dhamija, Tashvik Balaji, Pranav Wang, Yaohui Dantcheva, Antitza |
| author_facet | Chopin, Baptiste Dhamija, Tashvik Balaji, Pranav Wang, Yaohui Dantcheva, Antitza |
| contents | We propose Dimitra, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we train a conditional Motion Diffusion Transformer (cMDT) by modeling facial motion sequences with 3D representation. We condition the cMDT with only two input signals, an audio-sequence, as well as a reference facial image. By extracting additional features directly from audio, Dimitra is able to increase quality and realism of generated videos. In particular, phoneme sequences contribute to the realism of lip motion, whereas text transcript to facial expression and head pose realism. Quantitative and qualitative experiments on two widely employed datasets, VoxCeleb2 and HDTF, showcase that Dimitra is able to outperform existing approaches for generating realistic talking heads imparting lip motion, facial expression, and head pose. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_17198 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation Chopin, Baptiste Dhamija, Tashvik Balaji, Pranav Wang, Yaohui Dantcheva, Antitza Computer Vision and Pattern Recognition We propose Dimitra, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we train a conditional Motion Diffusion Transformer (cMDT) by modeling facial motion sequences with 3D representation. We condition the cMDT with only two input signals, an audio-sequence, as well as a reference facial image. By extracting additional features directly from audio, Dimitra is able to increase quality and realism of generated videos. In particular, phoneme sequences contribute to the realism of lip motion, whereas text transcript to facial expression and head pose realism. Quantitative and qualitative experiments on two widely employed datasets, VoxCeleb2 and HDTF, showcase that Dimitra is able to outperform existing approaches for generating realistic talking heads imparting lip motion, facial expression, and head pose. |
| title | Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.17198 |