ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915940608770048 |
|---|---|
| author | Yang, Jianxuan Guo, Xinyue Cheng, Zhi Wang, Kai Zhang, Lipan Hu, Jinjie Ji, Qiang Cao, Yihua Meng, Yihao Cui, Zhaoyue Liu, Mengmei Meng, Meng Luan, Jian |
| author_facet | Yang, Jianxuan Guo, Xinyue Cheng, Zhi Wang, Kai Zhang, Lipan Hu, Jinjie Ji, Qiang Cao, Yihua Meng, Yihao Cui, Zhaoyue Liu, Mengmei Meng, Meng Luan, Jian |
| contents | Recent advances in video-to-audio (V2A) generation enable high-quality audio synthesis from visual content, yet achieving robust and fine-grained controllability remains challenging. Existing methods suffer from weak textual controllability under visual-text conflict and imprecise stylistic control due to entangled temporal and timbre information in reference audio. Moreover, the lack of standardized benchmarks limits systematic evaluation.
We propose ControlFoley, a unified multimodal V2A framework that enables precise control over video, text, and reference audio. We introduce a joint visual encoding paradigm that integrates CLIP with a spatio-temporal audio-visual encoder to improve alignment and textual controllability. We further propose temporal-timbre decoupling to suppress redundant temporal cues while preserving discriminative timbre features. In addition, we design a modality-robust training scheme with unified multimodal representation alignment (REPA) and random modality dropout. We also present VGGSound-TVC, a benchmark for evaluating textual controllability under varying degrees of visual-text conflict.
Extensive experiments demonstrate state-of-the-art performance across multiple V2A tasks, including text-guided, text-controlled, and audio-controlled generation. ControlFoley achieves superior controllability under cross-modal conflict while maintaining strong synchronization and audio quality, and shows competitive or better performance compared to an industrial V2A system.
Code, models, datasets, and demos are available at: https://yjx-research.github.io/ControlFoley/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_15086 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling Yang, Jianxuan Guo, Xinyue Cheng, Zhi Wang, Kai Zhang, Lipan Hu, Jinjie Ji, Qiang Cao, Yihua Meng, Yihao Cui, Zhaoyue Liu, Mengmei Meng, Meng Luan, Jian Multimedia Computer Vision and Pattern Recognition Sound Recent advances in video-to-audio (V2A) generation enable high-quality audio synthesis from visual content, yet achieving robust and fine-grained controllability remains challenging. Existing methods suffer from weak textual controllability under visual-text conflict and imprecise stylistic control due to entangled temporal and timbre information in reference audio. Moreover, the lack of standardized benchmarks limits systematic evaluation. We propose ControlFoley, a unified multimodal V2A framework that enables precise control over video, text, and reference audio. We introduce a joint visual encoding paradigm that integrates CLIP with a spatio-temporal audio-visual encoder to improve alignment and textual controllability. We further propose temporal-timbre decoupling to suppress redundant temporal cues while preserving discriminative timbre features. In addition, we design a modality-robust training scheme with unified multimodal representation alignment (REPA) and random modality dropout. We also present VGGSound-TVC, a benchmark for evaluating textual controllability under varying degrees of visual-text conflict. Extensive experiments demonstrate state-of-the-art performance across multiple V2A tasks, including text-guided, text-controlled, and audio-controlled generation. ControlFoley achieves superior controllability under cross-modal conflict while maintaining strong synchronization and audio quality, and shows competitive or better performance compared to an industrial V2A system. Code, models, datasets, and demos are available at: https://yjx-research.github.io/ControlFoley/. |
| title | ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling |
| topic | Multimedia Computer Vision and Pattern Recognition Sound |
| url | https://arxiv.org/abs/2604.15086 |