Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.01952 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910554835124224 |
|---|---|
| author | He, Xiang Liu, Xiangxi Li, Yang Zhao, Dongcheng Shen, Guobin Kong, Qingqun Yang, Xin Zeng, Yi |
| author_facet | He, Xiang Liu, Xiangxi Li, Yang Zhao, Dongcheng Shen, Guobin Kong, Qingqun Yang, Xin Zeng, Yi |
| contents | The audio-visual event localization task requires identifying concurrent visual and auditory events from unconstrained videos within a network model, locating them, and classifying their category. The efficient extraction and integration of audio and visual modal information have always been challenging in this field. In this paper, we introduce CACE-Net, which differs from most existing methods that solely use audio signals to guide visual information. We propose an audio-visual co-guidance attention mechanism that allows for adaptive bi-directional cross-modal attentional guidance between audio and visual information, thus reducing inconsistencies between modalities. Moreover, we have observed that existing methods have difficulty distinguishing between similar background and event and lack the fine-grained features for event classification. Consequently, we employ background-event contrast enhancement to increase the discrimination of fused feature and fine-tuned pre-trained model to extract more refined and discernible features from complex multimodal inputs. Specifically, we have enhanced the model's ability to discern subtle differences between event and background and improved the accuracy of event classification in our model. Experiments on the AVE dataset demonstrate that CACE-Net sets a new benchmark in the audio-visual event localization task, proving the effectiveness of our proposed methods in handling complex multimodal learning and event localization in unconstrained videos. Code is available at https://github.com/Brain-Cog-Lab/CACE-Net. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01952 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization He, Xiang Liu, Xiangxi Li, Yang Zhao, Dongcheng Shen, Guobin Kong, Qingqun Yang, Xin Zeng, Yi Computer Vision and Pattern Recognition The audio-visual event localization task requires identifying concurrent visual and auditory events from unconstrained videos within a network model, locating them, and classifying their category. The efficient extraction and integration of audio and visual modal information have always been challenging in this field. In this paper, we introduce CACE-Net, which differs from most existing methods that solely use audio signals to guide visual information. We propose an audio-visual co-guidance attention mechanism that allows for adaptive bi-directional cross-modal attentional guidance between audio and visual information, thus reducing inconsistencies between modalities. Moreover, we have observed that existing methods have difficulty distinguishing between similar background and event and lack the fine-grained features for event classification. Consequently, we employ background-event contrast enhancement to increase the discrimination of fused feature and fine-tuned pre-trained model to extract more refined and discernible features from complex multimodal inputs. Specifically, we have enhanced the model's ability to discern subtle differences between event and background and improved the accuracy of event classification in our model. Experiments on the AVE dataset demonstrate that CACE-Net sets a new benchmark in the audio-visual event localization task, proving the effectiveness of our proposed methods in handling complex multimodal learning and event localization in unconstrained videos. Code is available at https://github.com/Brain-Cog-Lab/CACE-Net. |
| title | CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.01952 |