Multimodal Fusion with Pre-Trained Model Features in Affective Behaviour Analysis In-the-wild
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910378575790080 |
|---|---|
| author | Wen, Zhuofan Zhang, Fengyu Zhang, Siyuan Sun, Haiyang Xu, Mingyu Sun, Licai Lian, Zheng Liu, Bin Tao, Jianhua |
| author_facet | Wen, Zhuofan Zhang, Fengyu Zhang, Siyuan Sun, Haiyang Xu, Mingyu Sun, Licai Lian, Zheng Liu, Bin Tao, Jianhua |
| contents | Multimodal fusion is a significant method for most multimodal tasks. With the recent surge in the number of large pre-trained models, combining both multimodal fusion methods and pre-trained model features can achieve outstanding performance in many multimodal tasks. In this paper, we present our approach, which leverages both advantages for addressing the task of Expression (Expr) Recognition and Valence-Arousal (VA) Estimation. We evaluate the Aff-Wild2 database using pre-trained models, then extract the final hidden layers of the models as features. Following preprocessing and interpolation or convolution to align the extracted features, different models are employed for modal fusion. Our code is available at GitHub - FulgenceWen/ABAW6th. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_15044 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multimodal Fusion with Pre-Trained Model Features in Affective Behaviour Analysis In-the-wild Wen, Zhuofan Zhang, Fengyu Zhang, Siyuan Sun, Haiyang Xu, Mingyu Sun, Licai Lian, Zheng Liu, Bin Tao, Jianhua Computer Vision and Pattern Recognition Artificial Intelligence Multimodal fusion is a significant method for most multimodal tasks. With the recent surge in the number of large pre-trained models, combining both multimodal fusion methods and pre-trained model features can achieve outstanding performance in many multimodal tasks. In this paper, we present our approach, which leverages both advantages for addressing the task of Expression (Expr) Recognition and Valence-Arousal (VA) Estimation. We evaluate the Aff-Wild2 database using pre-trained models, then extract the final hidden layers of the models as features. Following preprocessing and interpolation or convolution to align the extracted features, different models are employed for modal fusion. Our code is available at GitHub - FulgenceWen/ABAW6th. |
| title | Multimodal Fusion with Pre-Trained Model Features in Affective Behaviour Analysis In-the-wild |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2403.15044 |