A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2022
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916313676382208 |
|---|---|
| author | Praveen, R. Gnana de Melo, Wheidima Carneiro Ullah, Nasib Aslam, Haseeb Zeeshan, Osama Denorme, Théo Pedersoli, Marco Koerich, Alessandro Bacon, Simon Cardinal, Patrick Granger, Eric |
| author_facet | Praveen, R. Gnana de Melo, Wheidima Carneiro Ullah, Nasib Aslam, Haseeb Zeeshan, Osama Denorme, Théo Pedersoli, Marco Koerich, Alessandro Bacon, Simon Cardinal, Patrick Granger, Eric |
| contents | Multimodal emotion recognition has recently gained much attention since it can leverage diverse and complementary relationships over multiple modalities (e.g., audio, visual, biosignals, etc.), and can provide some robustness to noisy modalities. Most state-of-the-art methods for audio-visual (A-V) fusion rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities. In this paper, we focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. Specifically, we propose a joint cross-attention model that relies on the complementary relationships to extract the salient features across A-V modalities, allowing for accurate prediction of continuous values of valence and arousal. The proposed fusion model efficiently leverages the inter-modal relationships, while reducing the heterogeneity between the features. In particular, it computes the cross-attention weights based on correlation between the combined feature representation and individual modalities. By deploying the combined A-V feature representation into the cross-attention module, the performance of our fusion module improves significantly over the vanilla cross-attention module. Experimental results on validation-set videos from the AffWild2 dataset indicate that our proposed A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches. The code is available on GitHub: https://github.com/praveena2j/JointCrossAttentional-AV-Fusion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2203_14779 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition Praveen, R. Gnana de Melo, Wheidima Carneiro Ullah, Nasib Aslam, Haseeb Zeeshan, Osama Denorme, Théo Pedersoli, Marco Koerich, Alessandro Bacon, Simon Cardinal, Patrick Granger, Eric Computer Vision and Pattern Recognition Human-Computer Interaction Sound Audio and Speech Processing Multimodal emotion recognition has recently gained much attention since it can leverage diverse and complementary relationships over multiple modalities (e.g., audio, visual, biosignals, etc.), and can provide some robustness to noisy modalities. Most state-of-the-art methods for audio-visual (A-V) fusion rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities. In this paper, we focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. Specifically, we propose a joint cross-attention model that relies on the complementary relationships to extract the salient features across A-V modalities, allowing for accurate prediction of continuous values of valence and arousal. The proposed fusion model efficiently leverages the inter-modal relationships, while reducing the heterogeneity between the features. In particular, it computes the cross-attention weights based on correlation between the combined feature representation and individual modalities. By deploying the combined A-V feature representation into the cross-attention module, the performance of our fusion module improves significantly over the vanilla cross-attention module. Experimental results on validation-set videos from the AffWild2 dataset indicate that our proposed A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches. The code is available on GitHub: https://github.com/praveena2j/JointCrossAttentional-AV-Fusion. |
| title | A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition |
| topic | Computer Vision and Pattern Recognition Human-Computer Interaction Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2203.14779 |