Joint Multimodal Transformer for Emotion Recognition in the Wild
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914762596548608 |
|---|---|
| author | Waligora, Paul Aslam, Haseeb Zeeshan, Osama Belharbi, Soufiane Koerich, Alessandro Lameiras Pedersoli, Marco Bacon, Simon Granger, Eric |
| author_facet | Waligora, Paul Aslam, Haseeb Zeeshan, Osama Belharbi, Soufiane Koerich, Alessandro Lameiras Pedersoli, Marco Bacon, Simon Granger, Eric |
| contents | Multimodal emotion recognition (MMER) systems typically outperform unimodal systems by leveraging the inter- and intra-modal relationships between, e.g., visual, textual, physiological, and auditory modalities. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention. This framework can exploit the complementary nature of diverse modalities to improve predictive accuracy. Separate backbones capture intra-modal spatiotemporal dependencies within each modality over video sequences. Subsequently, our JMT fusion architecture integrates the individual modality embeddings, allowing the model to effectively capture inter- and intra-modal relationships. Extensive experiments on two challenging expression recognition tasks -- (1) dimensional emotion recognition on the Affwild2 dataset (with face and voice) and (2) pain estimation on the Biovid dataset (with face and biosensors) -- indicate that our JMT fusion can provide a cost-effective solution for MMER. Empirical results show that MMER systems with our proposed fusion allow us to outperform relevant baseline and state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_10488 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Joint Multimodal Transformer for Emotion Recognition in the Wild Waligora, Paul Aslam, Haseeb Zeeshan, Osama Belharbi, Soufiane Koerich, Alessandro Lameiras Pedersoli, Marco Bacon, Simon Granger, Eric Computer Vision and Pattern Recognition Machine Learning Sound Audio and Speech Processing Multimodal emotion recognition (MMER) systems typically outperform unimodal systems by leveraging the inter- and intra-modal relationships between, e.g., visual, textual, physiological, and auditory modalities. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention. This framework can exploit the complementary nature of diverse modalities to improve predictive accuracy. Separate backbones capture intra-modal spatiotemporal dependencies within each modality over video sequences. Subsequently, our JMT fusion architecture integrates the individual modality embeddings, allowing the model to effectively capture inter- and intra-modal relationships. Extensive experiments on two challenging expression recognition tasks -- (1) dimensional emotion recognition on the Affwild2 dataset (with face and voice) and (2) pain estimation on the Biovid dataset (with face and biosensors) -- indicate that our JMT fusion can provide a cost-effective solution for MMER. Empirical results show that MMER systems with our proposed fusion allow us to outperform relevant baseline and state-of-the-art methods. |
| title | Joint Multimodal Transformer for Emotion Recognition in the Wild |
| topic | Computer Vision and Pattern Recognition Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2403.10488 |