Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.25066 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915885462061056 |
|---|---|
| author | He, Xu Zhang, Haoxian Chen, Hejia Zheng, Changyuan Chen, Liyang Tang, Songlin Huang, Jiehui Liu, Xiaoqiang Wan, Pengfei Wu, Zhiyong |
| author_facet | He, Xu Zhang, Haoxian Chen, Hejia Zheng, Changyuan Chen, Liyang Tang, Songlin Huang, Jiehui Liu, Xiaoqiang Wan, Pengfei Wu, Zhiyong |
| contents | Audio-driven visual dubbing aims to synchronize a video's lip movements with new speech but is fundamentally challenged by the lack of ideal training data: paired videos differing only in lip motion. Existing methods circumvent this via mask-based inpainting. However, masking inevitably destroys spatiotemporal context, leading to identity drift and poor robustness (e.g., to occlusions), while also inducing lip-shape leakage that degrades lip sync. To bridge this gap, we propose X-Dub, a novel two-stage generative bootstrapping framework leveraging powerful Diffusion Transformers to unlock mask-free dubbing. Our core insight is to repurpose a mask-based inpainting model exclusively as a dedicated data generator to synthesize scalable, high-fidelity pseudo-paired data, which is subsequently utilized to train and bootstrap a robust, mask-free editing model as the final video dubber. The final dubber is liberated from masking artifacts and leverages the complete video input for high-fidelity inference. We further introduce timestep-adaptive multi-phase learning to disentangle conflicting objectives (structure, lip motion, and texture) across diffusion phases, facilitating stable convergence and advanced editing quality. Additionally, we present X-DubBench, a benchmark for diverse scenarios. Extensive experiments demonstrate that our method achieves state-of-the-art performance with superior lip sync, visual quality, and robustness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_25066 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Inpainting to Editing: Unlocking Robust Mask-Free Visual Dubbing via Generative Bootstrapping He, Xu Zhang, Haoxian Chen, Hejia Zheng, Changyuan Chen, Liyang Tang, Songlin Huang, Jiehui Liu, Xiaoqiang Wan, Pengfei Wu, Zhiyong Computer Vision and Pattern Recognition Audio-driven visual dubbing aims to synchronize a video's lip movements with new speech but is fundamentally challenged by the lack of ideal training data: paired videos differing only in lip motion. Existing methods circumvent this via mask-based inpainting. However, masking inevitably destroys spatiotemporal context, leading to identity drift and poor robustness (e.g., to occlusions), while also inducing lip-shape leakage that degrades lip sync. To bridge this gap, we propose X-Dub, a novel two-stage generative bootstrapping framework leveraging powerful Diffusion Transformers to unlock mask-free dubbing. Our core insight is to repurpose a mask-based inpainting model exclusively as a dedicated data generator to synthesize scalable, high-fidelity pseudo-paired data, which is subsequently utilized to train and bootstrap a robust, mask-free editing model as the final video dubber. The final dubber is liberated from masking artifacts and leverages the complete video input for high-fidelity inference. We further introduce timestep-adaptive multi-phase learning to disentangle conflicting objectives (structure, lip motion, and texture) across diffusion phases, facilitating stable convergence and advanced editing quality. Additionally, we present X-DubBench, a benchmark for diverse scenarios. Extensive experiments demonstrate that our method achieves state-of-the-art performance with superior lip sync, visual quality, and robustness. |
| title | From Inpainting to Editing: Unlocking Robust Mask-Free Visual Dubbing via Generative Bootstrapping |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.25066 |