LINK: Adaptive Modality Interaction for Audio-Visual Video Parsing
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917881928744960 |
|---|---|
| author | Wang, Langyu Zhu, Bingke Chen, Yingying Wang, Jinqiao |
| author_facet | Wang, Langyu Zhu, Bingke Chen, Yingying Wang, Jinqiao |
| contents | Audio-visual video parsing focuses on classifying videos through weak labels while identifying events as either visible, audible, or both, alongside their respective temporal boundaries. Many methods ignore that different modalities often lack alignment, thereby introducing extra noise during modal interaction. In this work, we introduce a Learning Interaction method for Non-aligned Knowledge (LINK), designed to equilibrate the contributions of distinct modalities by dynamically adjusting their input during event prediction. Additionally, we leverage the semantic information of pseudo-labels as a priori knowledge to mitigate noise from other modalities. Our experimental findings demonstrate that our model outperforms existing methods on the LLP dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_20872 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LINK: Adaptive Modality Interaction for Audio-Visual Video Parsing Wang, Langyu Zhu, Bingke Chen, Yingying Wang, Jinqiao Computer Vision and Pattern Recognition Audio-visual video parsing focuses on classifying videos through weak labels while identifying events as either visible, audible, or both, alongside their respective temporal boundaries. Many methods ignore that different modalities often lack alignment, thereby introducing extra noise during modal interaction. In this work, we introduce a Learning Interaction method for Non-aligned Knowledge (LINK), designed to equilibrate the contributions of distinct modalities by dynamically adjusting their input during event prediction. Additionally, we leverage the semantic information of pseudo-labels as a priori knowledge to mitigate noise from other modalities. Our experimental findings demonstrate that our model outperforms existing methods on the LLP dataset. |
| title | LINK: Adaptive Modality Interaction for Audio-Visual Video Parsing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.20872 |