Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.10523 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915196287582208 |
|---|---|
| author | Yu, Jun Wang, Yongqi Wang, Lei Zheng, Yang Xu, Shengfan |
| author_facet | Yu, Jun Wang, Yongqi Wang, Lei Zheng, Yang Xu, Shengfan |
| contents | This paper presents our method for the estimation of valence-arousal (VA) in the 8th Affective Behavior Analysis in-the-Wild (ABAW) competition. Our approach integrates visual and audio information through a multimodal framework. The visual branch uses a pre-trained ResNet model to extract spatial features from facial images. The audio branches employ pre-trained VGG models to extract VGGish and LogMel features from speech signals. These features undergo temporal modeling using Temporal Convolutional Networks (TCNs). We then apply cross-modal attention mechanisms, where visual features interact with audio features through query-key-value attention structures. Finally, the features are concatenated and passed through a regression layer to predict valence and arousal. Our method achieves competitive performance on the Aff-Wild2 dataset, demonstrating effective multimodal fusion for VA estimation in-the-wild. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10523 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Interactive Multimodal Fusion with Temporal Modeling Yu, Jun Wang, Yongqi Wang, Lei Zheng, Yang Xu, Shengfan Computer Vision and Pattern Recognition This paper presents our method for the estimation of valence-arousal (VA) in the 8th Affective Behavior Analysis in-the-Wild (ABAW) competition. Our approach integrates visual and audio information through a multimodal framework. The visual branch uses a pre-trained ResNet model to extract spatial features from facial images. The audio branches employ pre-trained VGG models to extract VGGish and LogMel features from speech signals. These features undergo temporal modeling using Temporal Convolutional Networks (TCNs). We then apply cross-modal attention mechanisms, where visual features interact with audio features through query-key-value attention structures. Finally, the features are concatenated and passed through a regression layer to predict valence and arousal. Our method achieves competitive performance on the Aff-Wild2 dataset, demonstrating effective multimodal fusion for VA estimation in-the-wild. |
| title | Interactive Multimodal Fusion with Temporal Modeling |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.10523 |