Invariant Representation Guided Multimodal Sentiment Decoding with Sequential Variation Regularization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911412482211840 |
|---|---|
| author | Xu, Guoyang Song, Zhenxi Xue, Junqi Liu, Yuxin Wang, Zirui Zhang, Zhiguo |
| author_facet | Xu, Guoyang Song, Zhenxi Xue, Junqi Liu, Yuxin Wang, Zirui Zhang, Zhiguo |
| contents | Achieving consistent sentiment representation across diverse modalities remains a key challenge in multimodal sentiment analysis. However, rapid emotional fluctuations over time often introduce instability, leading to compromised prediction performance. To address this challenge, we propose a robust sentiment representation dual enhancement strategy that simultaneously enhances the temporal and modality dimensions, guided by targeted mechanisms in both forward and backward propagation. Specifically, in the modality dimension, we introduce a modality invariant fusion mechanism that fosters stable cross-modal representations, which aim to capture the common and stable representations shared across different modalities. In the temporal dimension, we impose a specialized sequential variation regularization term that regulates the model's learning trajectory during backward propagation, which is essentially total variation regularization degenerated into one-dimensional linear differences. Extensive experiments on three standard public datasets validate the effectiveness of our proposed approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00143 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Invariant Representation Guided Multimodal Sentiment Decoding with Sequential Variation Regularization Xu, Guoyang Song, Zhenxi Xue, Junqi Liu, Yuxin Wang, Zirui Zhang, Zhiguo Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Achieving consistent sentiment representation across diverse modalities remains a key challenge in multimodal sentiment analysis. However, rapid emotional fluctuations over time often introduce instability, leading to compromised prediction performance. To address this challenge, we propose a robust sentiment representation dual enhancement strategy that simultaneously enhances the temporal and modality dimensions, guided by targeted mechanisms in both forward and backward propagation. Specifically, in the modality dimension, we introduce a modality invariant fusion mechanism that fosters stable cross-modal representations, which aim to capture the common and stable representations shared across different modalities. In the temporal dimension, we impose a specialized sequential variation regularization term that regulates the model's learning trajectory during backward propagation, which is essentially total variation regularization degenerated into one-dimensional linear differences. Extensive experiments on three standard public datasets validate the effectiveness of our proposed approach. |
| title | Invariant Representation Guided Multimodal Sentiment Decoding with Sequential Variation Regularization |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.00143 |