StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916308389462016 |
|---|---|
| author | Cong, Gaoxiang Qi, Yuankai Li, Liang Beheshti, Amin Zhang, Zhedong Hengel, Anton van den Yang, Ming-Hsuan Yan, Chenggang Huang, Qingming |
| author_facet | Cong, Gaoxiang Qi, Yuankai Li, Liang Beheshti, Amin Zhang, Zhedong Hengel, Anton van den Yang, Ming-Hsuan Yan, Chenggang Huang, Qingming |
| contents | Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing state-of-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose StyleDubber, which switches dubbing learning from the frame level to phoneme level. It contains three main components: (1) A multimodal style adaptor operating at the phoneme level to learn pronunciation style from the reference audio, and generate intermediate representations informed by the facial emotion presented in the video; (2) An utterance-level style learning module, which guides both the mel-spectrogram decoding and the refining processes from the intermediate embeddings to improve the overall style expression; And (3) a phoneme-guided lip aligner to maintain lip sync. Extensive experiments on two of the primary benchmarks, V2C and Grid, demonstrate the favorable performance of the proposed method as compared to the current stateof-the-art. The code will be made available at https://github.com/GalaxyCong/StyleDubber. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_12636 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing Cong, Gaoxiang Qi, Yuankai Li, Liang Beheshti, Amin Zhang, Zhedong Hengel, Anton van den Yang, Ming-Hsuan Yan, Chenggang Huang, Qingming Computation and Language Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing state-of-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose StyleDubber, which switches dubbing learning from the frame level to phoneme level. It contains three main components: (1) A multimodal style adaptor operating at the phoneme level to learn pronunciation style from the reference audio, and generate intermediate representations informed by the facial emotion presented in the video; (2) An utterance-level style learning module, which guides both the mel-spectrogram decoding and the refining processes from the intermediate embeddings to improve the overall style expression; And (3) a phoneme-guided lip aligner to maintain lip sync. Extensive experiments on two of the primary benchmarks, V2C and Grid, demonstrate the favorable performance of the proposed method as compared to the current stateof-the-art. The code will be made available at https://github.com/GalaxyCong/StyleDubber. |
| title | StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.12636 |