Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918037196636160 |
|---|---|
| author | Xu, Le Li, Chenxing Ren, Yong Chen, Yujie Gu, Yu Fu, Ruibo Yang, Shan Yu, Dong |
| author_facet | Xu, Le Li, Chenxing Ren, Yong Chen, Yujie Gu, Yu Fu, Ruibo Yang, Shan Yu, Dong |
| contents | Current vision-guided audio captioning systems frequently fail to address audiovisual misalignment in real-world scenarios, such as dubbed content or off-screen sounds. To bridge this critical gap, we present an entropy-aware gated fusion framework that dynamically modulates visual information flow through cross-modal uncertainty quantification. Our novel approach employs attention entropy analysis in cross-attention layers to automatically identify and suppress misleading visual cues during modal fusion. Complementing this architecture, we develop a batch-wise audiovisual shuffling technique that generates synthetic mismatched training pairs, greatly enhancing model resilience against alignment noise. Evaluations on the AudioCaps benchmark demonstrate our system's superior performance over existing baselines, especially in mismatched modality scenarios. Furthermore, our solution demonstrates an approximately 6x improvement in inference speed compared to the baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22045 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning Xu, Le Li, Chenxing Ren, Yong Chen, Yujie Gu, Yu Fu, Ruibo Yang, Shan Yu, Dong Multimedia Computer Vision and Pattern Recognition Sound Audio and Speech Processing Current vision-guided audio captioning systems frequently fail to address audiovisual misalignment in real-world scenarios, such as dubbed content or off-screen sounds. To bridge this critical gap, we present an entropy-aware gated fusion framework that dynamically modulates visual information flow through cross-modal uncertainty quantification. Our novel approach employs attention entropy analysis in cross-attention layers to automatically identify and suppress misleading visual cues during modal fusion. Complementing this architecture, we develop a batch-wise audiovisual shuffling technique that generates synthetic mismatched training pairs, greatly enhancing model resilience against alignment noise. Evaluations on the AudioCaps benchmark demonstrate our system's superior performance over existing baselines, especially in mismatched modality scenarios. Furthermore, our solution demonstrates an approximately 6x improvement in inference speed compared to the baseline. |
| title | Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning |
| topic | Multimedia Computer Vision and Pattern Recognition Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.22045 |