Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.24550 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910222891614208 |
|---|---|
| author | Grassucci, Eleonora Galadini, Giuliano Cicchetti, Giordano Uncini, Aurelio Antonacci, Fabio Comminiello, Danilo |
| author_facet | Grassucci, Eleonora Galadini, Giuliano Cicchetti, Giordano Uncini, Aurelio Antonacci, Fabio Comminiello, Danilo |
| contents | Video-to-audio (V2A) generation aims to synthesize realistic and semantically aligned audio from silent videos, with potential applications in video editing, Foley sound design, and assistive multimedia. Although the excellent results, existing approaches either require costly joint training on large-scale paired datasets or rely on pairwise similarities that may fail to capture global multimodal coherence. In this work, we propose a novel training-free multimodal guidance mechanism for V2A diffusion that leverages the volume spanned by the modality embeddings to enforce unified alignment across video, audio, and text. The proposed multimodal diffusion guidance (MDG) provides a lightweight, plug-and-play control signal that can be applied on top of any pretrained audio diffusion model without retraining. Experiments on VGGSound and AudioCaps demonstrate that our MDG consistently improves perceptual quality and multimodal alignment compared to baselines, proving the effectiveness of a joint multimodal guidance for V2A. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24550 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Training-Free Multimodal Guidance for Video to Audio Generation Grassucci, Eleonora Galadini, Giuliano Cicchetti, Giordano Uncini, Aurelio Antonacci, Fabio Comminiello, Danilo Machine Learning Sound Video-to-audio (V2A) generation aims to synthesize realistic and semantically aligned audio from silent videos, with potential applications in video editing, Foley sound design, and assistive multimedia. Although the excellent results, existing approaches either require costly joint training on large-scale paired datasets or rely on pairwise similarities that may fail to capture global multimodal coherence. In this work, we propose a novel training-free multimodal guidance mechanism for V2A diffusion that leverages the volume spanned by the modality embeddings to enforce unified alignment across video, audio, and text. The proposed multimodal diffusion guidance (MDG) provides a lightweight, plug-and-play control signal that can be applied on top of any pretrained audio diffusion model without retraining. Experiments on VGGSound and AudioCaps demonstrate that our MDG consistently improves perceptual quality and multimodal alignment compared to baselines, proving the effectiveness of a joint multimodal guidance for V2A. |
| title | Training-Free Multimodal Guidance for Video to Audio Generation |
| topic | Machine Learning Sound |
| url | https://arxiv.org/abs/2509.24550 |