CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911591906148352 |
|---|---|
| author | Cong, Gaoxiang Li, Liang Ye, Jiaxin Zhang, Zhedong Shan, Hongming Qi, Yuankai Huang, Qingming |
| author_facet | Cong, Gaoxiang Li, Liang Ye, Jiaxin Zhang, Zhedong Shan, Hongming Qi, Yuankai Huang, Qingming |
| contents | Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_12292 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing Cong, Gaoxiang Li, Liang Ye, Jiaxin Zhang, Zhedong Shan, Hongming Qi, Yuankai Huang, Qingming Sound Computer Vision and Pattern Recognition Multimedia Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics. |
| title | CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing |
| topic | Sound Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2604.12292 |