Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.01798 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913634379104256 |
|---|---|
| author | Wang, Qili Wu, Dajiang Xu, Zihang Huang, Junshi Lv, Jun |
| author_facet | Wang, Qili Wu, Dajiang Xu, Zihang Huang, Junshi Lv, Jun |
| contents | Significant progress has been made in talking-face video generation research; however, precise lip-audio synchronization and high visual quality remain challenging in editing lip shapes based on input audio. This paper introduces JoyGen, a novel two-stage framework for talking-face generation, comprising audio-driven lip motion generation and visual appearance synthesis. In the first stage, a 3D reconstruction model and an audio2motion model predict identity and expression coefficients respectively. Next, by integrating audio features with a facial depth map, we provide comprehensive supervision for precise lip-audio synchronization in facial generation. Additionally, we constructed a Chinese talking-face dataset containing 130 hours of high-quality video. JoyGen is trained on the open-source HDTF dataset and our curated dataset. Experimental results demonstrate superior lip-audio synchronization and visual quality achieved by our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_01798 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing Wang, Qili Wu, Dajiang Xu, Zihang Huang, Junshi Lv, Jun Computer Vision and Pattern Recognition Significant progress has been made in talking-face video generation research; however, precise lip-audio synchronization and high visual quality remain challenging in editing lip shapes based on input audio. This paper introduces JoyGen, a novel two-stage framework for talking-face generation, comprising audio-driven lip motion generation and visual appearance synthesis. In the first stage, a 3D reconstruction model and an audio2motion model predict identity and expression coefficients respectively. Next, by integrating audio features with a facial depth map, we provide comprehensive supervision for precise lip-audio synchronization in facial generation. Additionally, we constructed a Chinese talking-face dataset containing 130 hours of high-quality video. JoyGen is trained on the open-source HDTF dataset and our curated dataset. Experimental results demonstrate superior lip-audio synchronization and visual quality achieved by our method. |
| title | JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.01798 |