KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908274021892096 |
|---|---|
| author | Bigata, Antoni Stypułkowski, Michał Mira, Rodrigo Bounareli, Stella Vougioukas, Konstantinos Landgraf, Zoe Drobyshev, Nikita Zieba, Maciej Petridis, Stavros Pantic, Maja |
| author_facet | Bigata, Antoni Stypułkowski, Michał Mira, Rodrigo Bounareli, Stella Vougioukas, Konstantinos Landgraf, Zoe Drobyshev, Nikita Zieba, Maciej Petridis, Stavros Pantic, Maja |
| contents | Current audio-driven facial animation methods achieve impressive results for short videos but suffer from error accumulation and identity drift when extended to longer durations. Existing methods attempt to mitigate this through external spatial control, increasing long-term consistency but compromising the naturalness of motion. We propose KeyFace, a novel two-stage diffusion-based framework, to address these issues. In the first stage, keyframes are generated at a low frame rate, conditioned on audio input and an identity frame, to capture essential facial expressions and movements over extended periods of time. In the second stage, an interpolation model fills in the gaps between keyframes, ensuring smooth transitions and temporal coherence. To further enhance realism, we incorporate continuous emotion representations and handle a wide range of non-speech vocalizations (NSVs), such as laughter and sighs. We also introduce two new evaluation metrics for assessing lip synchronization and NSV generation. Experimental results show that KeyFace outperforms state-of-the-art methods in generating natural, coherent facial animations over extended durations, successfully encompassing NSVs and continuous emotions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01715 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation Bigata, Antoni Stypułkowski, Michał Mira, Rodrigo Bounareli, Stella Vougioukas, Konstantinos Landgraf, Zoe Drobyshev, Nikita Zieba, Maciej Petridis, Stavros Pantic, Maja Computer Vision and Pattern Recognition Artificial Intelligence Current audio-driven facial animation methods achieve impressive results for short videos but suffer from error accumulation and identity drift when extended to longer durations. Existing methods attempt to mitigate this through external spatial control, increasing long-term consistency but compromising the naturalness of motion. We propose KeyFace, a novel two-stage diffusion-based framework, to address these issues. In the first stage, keyframes are generated at a low frame rate, conditioned on audio input and an identity frame, to capture essential facial expressions and movements over extended periods of time. In the second stage, an interpolation model fills in the gaps between keyframes, ensuring smooth transitions and temporal coherence. To further enhance realism, we incorporate continuous emotion representations and handle a wide range of non-speech vocalizations (NSVs), such as laughter and sighs. We also introduce two new evaluation metrics for assessing lip synchronization and NSV generation. Experimental results show that KeyFace outperforms state-of-the-art methods in generating natural, coherent facial animations over extended durations, successfully encompassing NSVs and continuous emotions. |
| title | KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2503.01715 |