ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913459930660864 |
|---|---|
| author | Guan, Jiazhi Xu, Zhiliang Zhou, Hang Wang, Kaisiyuan He, Shengyi Zhang, Zhanwang Liang, Borong Feng, Haocheng Ding, Errui Liu, Jingtuo Wang, Jingdong Zhao, Youjian Liu, Ziwei |
| author_facet | Guan, Jiazhi Xu, Zhiliang Zhou, Hang Wang, Kaisiyuan He, Shengyi Zhang, Zhanwang Liang, Borong Feng, Haocheng Ding, Errui Liu, Jingtuo Wang, Jingdong Zhao, Youjian Liu, Ziwei |
| contents | Lip-syncing videos with given audio is the foundation for various applications including the creation of virtual presenters or performers. While recent studies explore high-fidelity lip-sync with different techniques, their task-orientated models either require long-term videos for clip-specific training or retain visible artifacts. In this paper, we propose a unified and effective framework ReSyncer, that synchronizes generalized audio-visual facial information. The key design is revisiting and rewiring the Style-based generator to efficiently adopt 3D facial dynamics predicted by a principled style-injected Transformer. By simply re-configuring the information insertion mechanisms within the noise and style space, our framework fuses motion and appearance with unified training. Extensive experiments demonstrate that ReSyncer not only produces high-fidelity lip-synced videos according to audio, but also supports multiple appealing properties that are suitable for creating virtual presenters and performers, including fast personalized fine-tuning, video-driven lip-syncing, the transfer of speaking styles, and even face swapping. Resources can be found at https://guanjz20.github.io/projects/ReSyncer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03284 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer Guan, Jiazhi Xu, Zhiliang Zhou, Hang Wang, Kaisiyuan He, Shengyi Zhang, Zhanwang Liang, Borong Feng, Haocheng Ding, Errui Liu, Jingtuo Wang, Jingdong Zhao, Youjian Liu, Ziwei Computer Vision and Pattern Recognition Graphics Multimedia Lip-syncing videos with given audio is the foundation for various applications including the creation of virtual presenters or performers. While recent studies explore high-fidelity lip-sync with different techniques, their task-orientated models either require long-term videos for clip-specific training or retain visible artifacts. In this paper, we propose a unified and effective framework ReSyncer, that synchronizes generalized audio-visual facial information. The key design is revisiting and rewiring the Style-based generator to efficiently adopt 3D facial dynamics predicted by a principled style-injected Transformer. By simply re-configuring the information insertion mechanisms within the noise and style space, our framework fuses motion and appearance with unified training. Extensive experiments demonstrate that ReSyncer not only produces high-fidelity lip-synced videos according to audio, but also supports multiple appealing properties that are suitable for creating virtual presenters and performers, including fast personalized fine-tuning, video-driven lip-syncing, the transfer of speaking styles, and even face swapping. Resources can be found at https://guanjz20.github.io/projects/ReSyncer. |
| title | ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performer |
| topic | Computer Vision and Pattern Recognition Graphics Multimedia |
| url | https://arxiv.org/abs/2408.03284 |