Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.22501 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908799107858432 |
|---|---|
| author | Lu, Renjie Zhang, Xulong Qu, Xiaoyang Wang, Jianzong Wang, Shangfei |
| author_facet | Lu, Renjie Zhang, Xulong Qu, Xiaoyang Wang, Jianzong Wang, Shangfei |
| contents | Synthesizing personalized talking faces that uphold and highlight a speaker's unique style while maintaining lip-sync accuracy remains a significant challenge. A primary limitation of existing approaches is the intrinsic confounding of speaker-specific talking style and semantic content within facial motions, which prevents the faithful transfer of a speaker's unique persona to arbitrary speech. In this paper, we propose MirrorTalk, a generative framework based on a conditional diffusion model, combined with a Semantically-Disentangled Style Encoder (SDSE) that can distill pure style representations from a brief reference video. To effectively utilize this representation, we further introduce a hierarchical modulation strategy within the diffusion process. This mechanism guides the synthesis by dynamically balancing the contributions of audio and style features across distinct facial regions, ensuring both precise lip-sync accuracy and expressive full-face dynamics. Extensive experiments demonstrate that MirrorTalk achieves significant improvements over state-of-the-art methods in terms of lip-sync accuracy and personalization preservation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22501 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MIRRORTALK: Forging Personalized Avatars Via Disentangled Style and Hierarchical Motion Control Lu, Renjie Zhang, Xulong Qu, Xiaoyang Wang, Jianzong Wang, Shangfei Computer Vision and Pattern Recognition Sound Synthesizing personalized talking faces that uphold and highlight a speaker's unique style while maintaining lip-sync accuracy remains a significant challenge. A primary limitation of existing approaches is the intrinsic confounding of speaker-specific talking style and semantic content within facial motions, which prevents the faithful transfer of a speaker's unique persona to arbitrary speech. In this paper, we propose MirrorTalk, a generative framework based on a conditional diffusion model, combined with a Semantically-Disentangled Style Encoder (SDSE) that can distill pure style representations from a brief reference video. To effectively utilize this representation, we further introduce a hierarchical modulation strategy within the diffusion process. This mechanism guides the synthesis by dynamically balancing the contributions of audio and style features across distinct facial regions, ensuring both precise lip-sync accuracy and expressive full-face dynamics. Extensive experiments demonstrate that MirrorTalk achieves significant improvements over state-of-the-art methods in terms of lip-sync accuracy and personalization preservation. |
| title | MIRRORTALK: Forging Personalized Avatars Via Disentangled Style and Hierarchical Motion Control |
| topic | Computer Vision and Pattern Recognition Sound |
| url | https://arxiv.org/abs/2601.22501 |