Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.02386 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912307166052352 |
|---|---|
| author | Sung-Bin, Kim Choi, Jeongsoo Peng, Puyuan Chung, Joon Son Oh, Tae-Hyun Harwath, David |
| author_facet | Sung-Bin, Kim Choi, Jeongsoo Peng, Puyuan Chung, Joon Son Oh, Tae-Hyun Harwath, David |
| contents | We present VoiceCraft-Dub, a novel approach for automated video dubbing that synthesizes high-quality speech from text and facial cues. This task has broad applications in filmmaking, multimedia creation, and assisting voice-impaired individuals. Building on the success of Neural Codec Language Models (NCLMs) for speech synthesis, our method extends their capabilities by incorporating video features, ensuring that synthesized speech is time-synchronized and expressively aligned with facial movements while preserving natural prosody. To inject visual cues, we design adapters to align facial features with the NCLM token space and introduce audio-visual fusion layers to merge audio-visual information within the NCLM framework. Additionally, we curate CelebV-Dub, a new dataset of expressive, real-world videos specifically designed for automated video dubbing. Extensive experiments show that our model achieves high-quality, intelligible, and natural speech synthesis with accurate lip synchronization, outperforming existing methods in human perception and performing favorably in objective evaluations. We also adapt VoiceCraft-Dub for the video-to-speech task, demonstrating its versatility for various applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_02386 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models Sung-Bin, Kim Choi, Jeongsoo Peng, Puyuan Chung, Joon Son Oh, Tae-Hyun Harwath, David Computer Vision and Pattern Recognition Audio and Speech Processing We present VoiceCraft-Dub, a novel approach for automated video dubbing that synthesizes high-quality speech from text and facial cues. This task has broad applications in filmmaking, multimedia creation, and assisting voice-impaired individuals. Building on the success of Neural Codec Language Models (NCLMs) for speech synthesis, our method extends their capabilities by incorporating video features, ensuring that synthesized speech is time-synchronized and expressively aligned with facial movements while preserving natural prosody. To inject visual cues, we design adapters to align facial features with the NCLM token space and introduce audio-visual fusion layers to merge audio-visual information within the NCLM framework. Additionally, we curate CelebV-Dub, a new dataset of expressive, real-world videos specifically designed for automated video dubbing. Extensive experiments show that our model achieves high-quality, intelligible, and natural speech synthesis with accurate lip synchronization, outperforming existing methods in human perception and performing favorably in objective evaluations. We also adapt VoiceCraft-Dub for the video-to-speech task, demonstrating its versatility for various applications. |
| title | VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models |
| topic | Computer Vision and Pattern Recognition Audio and Speech Processing |
| url | https://arxiv.org/abs/2504.02386 |