Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.12853 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909233753096192 |
|---|---|
| author | Rajasekhar, G Alam, Jahangir |
| author_facet | Rajasekhar, G Alam, Jahangir |
| contents | Leveraging complementary relationships across modalities has recently drawn a lot of attention in multimodal emotion recognition. Most of the existing approaches explored cross-attention to capture the complementary relationships across the modalities. However, the modalities may also exhibit weak complementary relationships, which may deteriorate the cross-attended features, resulting in poor multimodal feature representations. To address this problem, we propose Inconsistency-Aware Cross-Attention (IACA), which can adaptively select the most relevant features on-the-fly based on the strong or weak complementary relationships across audio and visual modalities. Specifically, we design a two-stage gating mechanism that can adaptively select the appropriate relevant features to deal with weak complementary relationships. Extensive experiments are conducted on the challenging Aff-Wild2 dataset to show the robustness of the proposed model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_12853 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Inconsistency-Aware Cross-Attention for Audio-Visual Fusion in Dimensional Emotion Recognition Rajasekhar, G Alam, Jahangir Computer Vision and Pattern Recognition Leveraging complementary relationships across modalities has recently drawn a lot of attention in multimodal emotion recognition. Most of the existing approaches explored cross-attention to capture the complementary relationships across the modalities. However, the modalities may also exhibit weak complementary relationships, which may deteriorate the cross-attended features, resulting in poor multimodal feature representations. To address this problem, we propose Inconsistency-Aware Cross-Attention (IACA), which can adaptively select the most relevant features on-the-fly based on the strong or weak complementary relationships across audio and visual modalities. Specifically, we design a two-stage gating mechanism that can adaptively select the appropriate relevant features to deal with weak complementary relationships. Extensive experiments are conducted on the challenging Aff-Wild2 dataset to show the robustness of the proposed model. |
| title | Inconsistency-Aware Cross-Attention for Audio-Visual Fusion in Dimensional Emotion Recognition |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.12853 |