Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918089928474624 |
|---|---|
| author | Liang, Yingshan Fan, Keyu Du, Zhicheng Wang, Yiran Shi, Qingyang Zhang, Xinyu Lu, Jiasheng Qin, Peiwu |
| author_facet | Liang, Yingshan Fan, Keyu Du, Zhicheng Wang, Yiran Shi, Qingyang Zhang, Xinyu Lu, Jiasheng Qin, Peiwu |
| contents | Video-to-audio (V2A) generation shows great potential in fields such as film production. Despite significant advances, current V2A methods relying on global video information struggle with complex scenes and generating audio tailored to specific objects. To address these limitations, we introduce Hear-Your-Click, an interactive V2A framework enabling users to generate sounds for specific objects by clicking on the frame. To achieve this, we propose Object-aware Contrastive Audio-Visual Fine-tuning (OCAV) with a Mask-guided Visual Encoder (MVE) to obtain object-level visual features aligned with audio. Furthermore, we tailor two data augmentation strategies, Random Video Stitching (RVS) and Mask-guided Loudness Modulation (MLM), to enhance the model's sensitivity to segmented objects. To measure audio-visual correspondence, we designed a new evaluation metric, the CAV score. Extensive experiments demonstrate that our framework offers more precise control and improves generation performance across various metrics. Project Page: https://github.com/SynapGrid/Hear-Your-Click |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04959 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation Liang, Yingshan Fan, Keyu Du, Zhicheng Wang, Yiran Shi, Qingyang Zhang, Xinyu Lu, Jiasheng Qin, Peiwu Computer Vision and Pattern Recognition Artificial Intelligence Multimedia Sound Audio and Speech Processing Video-to-audio (V2A) generation shows great potential in fields such as film production. Despite significant advances, current V2A methods relying on global video information struggle with complex scenes and generating audio tailored to specific objects. To address these limitations, we introduce Hear-Your-Click, an interactive V2A framework enabling users to generate sounds for specific objects by clicking on the frame. To achieve this, we propose Object-aware Contrastive Audio-Visual Fine-tuning (OCAV) with a Mask-guided Visual Encoder (MVE) to obtain object-level visual features aligned with audio. Furthermore, we tailor two data augmentation strategies, Random Video Stitching (RVS) and Mask-guided Loudness Modulation (MLM), to enhance the model's sensitivity to segmented objects. To measure audio-visual correspondence, we designed a new evaluation metric, the CAV score. Extensive experiments demonstrate that our framework offers more precise control and improves generation performance across various metrics. Project Page: https://github.com/SynapGrid/Hear-Your-Click |
| title | Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Multimedia Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2507.04959 |