See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915826650578944 |
|---|---|
| author | Wang, Jinting Wang, Jun Cheng, Hei Victor Liu, Li |
| author_facet | Wang, Jinting Wang, Jun Cheng, Hei Victor Liu, Li |
| contents | Unlike existing methods that rely on source images as appearance references and use source speech to generate motion, this work proposes a novel approach that directly extracts information from the speech, addressing key challenges in speech-to-talking face. Specifically, we first employ a speech-to-face portrait generation stage, utilizing a speech-conditioned diffusion model combined with statistical facial prior and a sample-adaptive weighting module to achieve high-quality portrait generation. In the subsequent speech-driven talking face generation stage, we embed expressive dynamics such as lip movement, facial expressions, and eye movements into the latent space of the diffusion model and further optimize lip synchronization using a region-enhancement module. To generate high-resolution outputs, we integrate a pre-trained Transformer-based discrete codebook with an image rendering network, enhancing video frame details in an end-to-end manner. Experimental results demonstrate that our method outperforms existing approaches on the HDTF, VoxCeleb, and AVSpeech datasets. Notably, this is the first method capable of generating high-resolution, high-quality talking face videos exclusively from a single speech input. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_26819 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement Wang, Jinting Wang, Jun Cheng, Hei Victor Liu, Li Audio and Speech Processing Artificial Intelligence Computer Vision and Pattern Recognition Sound Unlike existing methods that rely on source images as appearance references and use source speech to generate motion, this work proposes a novel approach that directly extracts information from the speech, addressing key challenges in speech-to-talking face. Specifically, we first employ a speech-to-face portrait generation stage, utilizing a speech-conditioned diffusion model combined with statistical facial prior and a sample-adaptive weighting module to achieve high-quality portrait generation. In the subsequent speech-driven talking face generation stage, we embed expressive dynamics such as lip movement, facial expressions, and eye movements into the latent space of the diffusion model and further optimize lip synchronization using a region-enhancement module. To generate high-resolution outputs, we integrate a pre-trained Transformer-based discrete codebook with an image rendering network, enhancing video frame details in an end-to-end manner. Experimental results demonstrate that our method outperforms existing approaches on the HDTF, VoxCeleb, and AVSpeech datasets. Notably, this is the first method capable of generating high-resolution, high-quality talking face videos exclusively from a single speech input. |
| title | See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement |
| topic | Audio and Speech Processing Artificial Intelligence Computer Vision and Pattern Recognition Sound |
| url | https://arxiv.org/abs/2510.26819 |