DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909688677793792 |
|---|---|
| author | Tian, Wenjie Zhu, Xinfa Liu, Haohe Zhao, Zhixian Chen, Zihao Ding, Chaofan Di, Xinhan Zheng, Junjie Xie, Lei |
| author_facet | Tian, Wenjie Zhu, Xinfa Liu, Haohe Zhao, Zhixian Chen, Zihao Ding, Chaofan Di, Xinhan Zheng, Junjie Xie, Lei |
| contents | While recent video-to-audio (V2A) models can generate realistic background audio from visual input, they largely overlook speech, an essential part of many video soundtracks. This paper proposes a new task, video-to-soundtrack (V2ST) generation, which aims to jointly produce synchronized background audio and speech within a unified framework. To tackle V2ST, we introduce DualDub, a unified framework built on a multimodal language model that integrates a multimodal encoder, a cross-modal aligner, and dual decoding heads for simultaneous background audio and speech generation. Specifically, our proposed cross-modal aligner employs causal and non-causal attention mechanisms to improve synchronization and acoustic harmony. Besides, to handle data scarcity, we design a curriculum learning strategy that progressively builds the multimodal capability. Finally, we introduce DualBench, the first benchmark for V2ST evaluation with a carefully curated test set and comprehensive metrics. Experimental results demonstrate that DualDub achieves state-of-the-art performance, generating high-quality and well-synchronized soundtracks with both speech and background audio. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_10109 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis Tian, Wenjie Zhu, Xinfa Liu, Haohe Zhao, Zhixian Chen, Zihao Ding, Chaofan Di, Xinhan Zheng, Junjie Xie, Lei Multimedia Sound Audio and Speech Processing While recent video-to-audio (V2A) models can generate realistic background audio from visual input, they largely overlook speech, an essential part of many video soundtracks. This paper proposes a new task, video-to-soundtrack (V2ST) generation, which aims to jointly produce synchronized background audio and speech within a unified framework. To tackle V2ST, we introduce DualDub, a unified framework built on a multimodal language model that integrates a multimodal encoder, a cross-modal aligner, and dual decoding heads for simultaneous background audio and speech generation. Specifically, our proposed cross-modal aligner employs causal and non-causal attention mechanisms to improve synchronization and acoustic harmony. Besides, to handle data scarcity, we design a curriculum learning strategy that progressively builds the multimodal capability. Finally, we introduce DualBench, the first benchmark for V2ST evaluation with a carefully curated test set and comprehensive metrics. Experimental results demonstrate that DualDub achieves state-of-the-art performance, generating high-quality and well-synchronized soundtracks with both speech and background audio. |
| title | DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis |
| topic | Multimedia Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2507.10109 |