VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912974695825408 |
|---|---|
| author | Cheng, Xin Wang, Yuyue Wang, Xihua Wu, Yihan Guan, Kaisi Chen, Yijing Zhang, Peng Liu, Xiaojiang Cao, Meng Song, Ruihua |
| author_facet | Cheng, Xin Wang, Yuyue Wang, Xihua Wu, Yihan Guan, Kaisi Chen, Yijing Zhang, Peng Liu, Xiaojiang Cao, Meng Song, Ruihua |
| contents | Video-conditioned audio generation, including Video-to-Sound (V2S) and Visual Text-to-Speech (VisualTTS), has traditionally been treated as distinct tasks, leaving the potential for a unified generative framework largely underexplored. In this paper, we bridge this gap with VSSFlow, a unified flow-matching framework that seamlessly solve both problems. To effectively handle multiple input signals within a Diffusion Transformer (DiT) architecture, we propose a disentangled condition aggregation mechanism leveraging distinct intrinsic properties of attention layers: cross-attention for semantic conditions, and self-attention for temporally-intensive conditions. Besides, contrary to the prevailing belief that joint training for the two tasks leads to performance degradation, we demonstrate that VSSFlow maintains superior performance during end-to-end joint learning process. Furthermore, we use a straightforward feature-level data synthesis method, demonstrating that our framework provides a robust foundation that easily adapts to joint sound and speech generation using synthetic data. Extensive experiments on V2S, VisualTTS and joint generation benchmarks show that VSSFlow effectively unifies these tasks and surpasses state-of-the-art domain-specific baselines, underscoring the critical potential of unified generative models. Project page: https://vasflow1.github.io/vasflow/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24773 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning Cheng, Xin Wang, Yuyue Wang, Xihua Wu, Yihan Guan, Kaisi Chen, Yijing Zhang, Peng Liu, Xiaojiang Cao, Meng Song, Ruihua Audio and Speech Processing Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Sound Video-conditioned audio generation, including Video-to-Sound (V2S) and Visual Text-to-Speech (VisualTTS), has traditionally been treated as distinct tasks, leaving the potential for a unified generative framework largely underexplored. In this paper, we bridge this gap with VSSFlow, a unified flow-matching framework that seamlessly solve both problems. To effectively handle multiple input signals within a Diffusion Transformer (DiT) architecture, we propose a disentangled condition aggregation mechanism leveraging distinct intrinsic properties of attention layers: cross-attention for semantic conditions, and self-attention for temporally-intensive conditions. Besides, contrary to the prevailing belief that joint training for the two tasks leads to performance degradation, we demonstrate that VSSFlow maintains superior performance during end-to-end joint learning process. Furthermore, we use a straightforward feature-level data synthesis method, demonstrating that our framework provides a robust foundation that easily adapts to joint sound and speech generation using synthetic data. Extensive experiments on V2S, VisualTTS and joint generation benchmarks show that VSSFlow effectively unifies these tasks and surpasses state-of-the-art domain-specific baselines, underscoring the critical potential of unified generative models. Project page: https://vasflow1.github.io/vasflow/ |
| title | VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Sound |
| url | https://arxiv.org/abs/2509.24773 |