Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911020555960320 |
|---|---|
| author | Wang, Jun Zeng, Xijuan Qiang, Chunyu Chen, Ruilong Wang, Shiyao Wang, Le Zhou, Wangjing Cai, Pengfei Zhao, Jiahui Li, Nan Li, Zihan Liang, Yuzhe Wang, Xiaopeng Zheng, Haorui Wen, Ming Yin, Kang Wang, Yiran Li, Nan Deng, Feng Dong, Liang Zhang, Chen Zhang, Di Gai, Kun |
| author_facet | Wang, Jun Zeng, Xijuan Qiang, Chunyu Chen, Ruilong Wang, Shiyao Wang, Le Zhou, Wangjing Cai, Pengfei Zhao, Jiahui Li, Nan Li, Zihan Liang, Yuzhe Wang, Xiaopeng Zheng, Haorui Wen, Ming Yin, Kang Wang, Yiran Li, Nan Deng, Feng Dong, Liang Zhang, Chen Zhang, Di Gai, Kun |
| contents | We propose Kling-Foley, a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content. In Kling-Foley, we introduce multimodal diffusion transformers to model the interactions between video, audio, and text modalities, and combine it with a visual semantic representation module and an audio-visual synchronization module to enhance alignment capabilities. Specifically, these modules align video conditions with latent audio elements at the frame level, thereby improving semantic alignment and audio-visual synchronization. Together with text conditions, this integrated approach enables precise generation of video-matching sound effects. In addition, we propose a universal latent audio codec that can achieve high-quality modeling in various scenarios such as sound effects, speech, singing, and music. We employ a stereo rendering method that imbues synthesized audio with a spatial presence. At the same time, in order to make up for the incomplete types and annotations of the open-source benchmark, we also open-source an industrial-level benchmark Kling-Audio-Eval. Our experiments show that Kling-Foley trained with the flow matching objective achieves new audio-visual SOTA performance among public models in terms of distribution matching, semantic alignment, temporal alignment and audio quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_19774 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation Wang, Jun Zeng, Xijuan Qiang, Chunyu Chen, Ruilong Wang, Shiyao Wang, Le Zhou, Wangjing Cai, Pengfei Zhao, Jiahui Li, Nan Li, Zihan Liang, Yuzhe Wang, Xiaopeng Zheng, Haorui Wen, Ming Yin, Kang Wang, Yiran Li, Nan Deng, Feng Dong, Liang Zhang, Chen Zhang, Di Gai, Kun Audio and Speech Processing Artificial Intelligence Computation and Language Sound We propose Kling-Foley, a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content. In Kling-Foley, we introduce multimodal diffusion transformers to model the interactions between video, audio, and text modalities, and combine it with a visual semantic representation module and an audio-visual synchronization module to enhance alignment capabilities. Specifically, these modules align video conditions with latent audio elements at the frame level, thereby improving semantic alignment and audio-visual synchronization. Together with text conditions, this integrated approach enables precise generation of video-matching sound effects. In addition, we propose a universal latent audio codec that can achieve high-quality modeling in various scenarios such as sound effects, speech, singing, and music. We employ a stereo rendering method that imbues synthesized audio with a spatial presence. At the same time, in order to make up for the incomplete types and annotations of the open-source benchmark, we also open-source an industrial-level benchmark Kling-Audio-Eval. Our experiments show that Kling-Foley trained with the flow matching objective achieves new audio-visual SOTA performance among public models in terms of distribution matching, semantic alignment, temporal alignment and audio quality. |
| title | Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Sound |
| url | https://arxiv.org/abs/2506.19774 |