Audio-driven Talking Face Generation with Stabilized Synchronization Loss
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910532499406848 |
|---|---|
| author | Yaman, Dogucan Eyiokur, Fevziye Irem Bärmann, Leonard Ekenel, Hazim Kemal Waibel, Alexander |
| author_facet | Yaman, Dogucan Eyiokur, Fevziye Irem Bärmann, Leonard Ekenel, Hazim Kemal Waibel, Alexander |
| contents | Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality, using given audio and reference video while preserving identity and visual characteristics. In this paper, we start by identifying several issues with existing synchronization learning methods. These involve unstable training, lip synchronization, and visual quality issues caused by lip-sync loss, SyncNet, and lip leaking from the identity reference. To address these issues, we first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. We then introduce stabilized synchronization loss and AVSyncNet to overcome problems caused by lip-sync loss and SyncNet. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization. Comprehensive ablation studies further validate our individual contributions and their cohesive effects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_09368 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Audio-driven Talking Face Generation with Stabilized Synchronization Loss Yaman, Dogucan Eyiokur, Fevziye Irem Bärmann, Leonard Ekenel, Hazim Kemal Waibel, Alexander Computer Vision and Pattern Recognition Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality, using given audio and reference video while preserving identity and visual characteristics. In this paper, we start by identifying several issues with existing synchronization learning methods. These involve unstable training, lip synchronization, and visual quality issues caused by lip-sync loss, SyncNet, and lip leaking from the identity reference. To address these issues, we first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. We then introduce stabilized synchronization loss and AVSyncNet to overcome problems caused by lip-sync loss and SyncNet. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization. Comprehensive ablation studies further validate our individual contributions and their cohesive effects. |
| title | Audio-driven Talking Face Generation with Stabilized Synchronization Loss |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2307.09368 |