Superior and Pragmatic Talking Face Generation with Teacher-Student Framework
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917622810935296 |
|---|---|
| author | Liang, Chao Jiang, Jianwen Zhong, Tianyun Lin, Gaojie Rong, Zhengkun Yang, Jiaqi Zhu, Yongming |
| author_facet | Liang, Chao Jiang, Jianwen Zhong, Tianyun Lin, Gaojie Rong, Zhengkun Yang, Jiaqi Zhu, Yongming |
| contents | Talking face generation technology creates talking videos from arbitrary appearance and motion signal, with the "arbitrary" offering ease of use but also introducing challenges in practical applications. Existing methods work well with standard inputs but suffer serious performance degradation with intricate real-world ones. Moreover, efficiency is also an important concern in deployment. To comprehensively address these issues, we introduce SuperFace, a teacher-student framework that balances quality, robustness, cost and editability. We first propose a simple but effective teacher model capable of handling inputs of varying qualities to generate high-quality results. Building on this, we devise an efficient distillation strategy to acquire an identity-specific student model that maintains quality with significantly reduced computational load. Our experiments validate that SuperFace offers a more comprehensive solution than existing methods for the four mentioned objectives, especially in reducing FLOPs by 99\% with the student model. SuperFace can be driven by both video and audio and allows for localized facial attributes editing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_17883 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Superior and Pragmatic Talking Face Generation with Teacher-Student Framework Liang, Chao Jiang, Jianwen Zhong, Tianyun Lin, Gaojie Rong, Zhengkun Yang, Jiaqi Zhu, Yongming Computer Vision and Pattern Recognition Talking face generation technology creates talking videos from arbitrary appearance and motion signal, with the "arbitrary" offering ease of use but also introducing challenges in practical applications. Existing methods work well with standard inputs but suffer serious performance degradation with intricate real-world ones. Moreover, efficiency is also an important concern in deployment. To comprehensively address these issues, we introduce SuperFace, a teacher-student framework that balances quality, robustness, cost and editability. We first propose a simple but effective teacher model capable of handling inputs of varying qualities to generate high-quality results. Building on this, we devise an efficient distillation strategy to acquire an identity-specific student model that maintains quality with significantly reduced computational load. Our experiments validate that SuperFace offers a more comprehensive solution than existing methods for the four mentioned objectives, especially in reducing FLOPs by 99\% with the student model. SuperFace can be driven by both video and audio and allows for localized facial attributes editing. |
| title | Superior and Pragmatic Talking Face Generation with Teacher-Student Framework |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.17883 |