Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911869965434880 |
|---|---|
| author | Yaman, Dogucan Eyiokur, Fevziye Irem Bärmann, Leonard Aktı, Seymanur Ekenel, Hazım Kemal Waibel, Alexander |
| author_facet | Yaman, Dogucan Eyiokur, Fevziye Irem Bärmann, Leonard Aktı, Seymanur Ekenel, Hazım Kemal Waibel, Alexander |
| contents | In the task of talking face generation, the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality, as well as robustly evaluating such synchronization. To tackle these problems, we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover, leveraging AV-HuBERT's features, we introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance. Experimental results, along with a detailed ablation study, demonstrate the effectiveness of our approach and the utility of the proposed evaluation metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_04327 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation Yaman, Dogucan Eyiokur, Fevziye Irem Bärmann, Leonard Aktı, Seymanur Ekenel, Hazım Kemal Waibel, Alexander Computer Vision and Pattern Recognition In the task of talking face generation, the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality, as well as robustly evaluating such synchronization. To tackle these problems, we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover, leveraging AV-HuBERT's features, we introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance. Experimental results, along with a detailed ablation study, demonstrate the effectiveness of our approach and the utility of the proposed evaluation metrics. |
| title | Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.04327 |