Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929576625569792 |
|---|---|
| author | Li, Mingcheng Yang, Dingkang Liu, Yang Wang, Shunli Chen, Jiawei Wang, Shuaibing Wei, Jinjie Jiang, Yue Xu, Qingyao Hou, Xiaolu Sun, Mingyang Qian, Ziyun Kou, Dongliang Zhang, Lihua |
| author_facet | Li, Mingcheng Yang, Dingkang Liu, Yang Wang, Shunli Chen, Jiawei Wang, Shuaibing Wei, Jinjie Jiang, Yue Xu, Qingyao Hou, Xiaolu Sun, Mingyang Qian, Ziyun Kou, Dongliang Zhang, Lihua |
| contents | Multimodal Sentiment Analysis (MSA) is an important research area that aims to understand and recognize human sentiment through multiple modalities. The complementary information provided by multimodal fusion promotes better sentiment analysis compared to utilizing only a single modality. Nevertheless, in real-world applications, many unavoidable factors may lead to situations of uncertain modality missing, thus hindering the effectiveness of multimodal modeling and degrading the model's performance. To this end, we propose a Hierarchical Representation Learning Framework (HRLF) for the MSA task under uncertain missing modalities. Specifically, we propose a fine-grained representation factorization module that sufficiently extracts valuable sentiment information by factorizing modality into sentiment-relevant and modality-specific representations through crossmodal translation and sentiment semantic reconstruction. Moreover, a hierarchical mutual information maximization mechanism is introduced to incrementally maximize the mutual information between multi-scale representations to align and reconstruct the high-level semantics in the representations. Ultimately, we propose a hierarchical adversarial learning mechanism that further aligns and adapts the latent distribution of sentiment-relevant representations to produce robust joint multimodal representations. Comprehensive experiments on three datasets demonstrate that HRLF significantly improves MSA performance under uncertain modality missing cases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_02793 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning Li, Mingcheng Yang, Dingkang Liu, Yang Wang, Shunli Chen, Jiawei Wang, Shuaibing Wei, Jinjie Jiang, Yue Xu, Qingyao Hou, Xiaolu Sun, Mingyang Qian, Ziyun Kou, Dongliang Zhang, Lihua Computation and Language Computer Vision and Pattern Recognition Multimodal Sentiment Analysis (MSA) is an important research area that aims to understand and recognize human sentiment through multiple modalities. The complementary information provided by multimodal fusion promotes better sentiment analysis compared to utilizing only a single modality. Nevertheless, in real-world applications, many unavoidable factors may lead to situations of uncertain modality missing, thus hindering the effectiveness of multimodal modeling and degrading the model's performance. To this end, we propose a Hierarchical Representation Learning Framework (HRLF) for the MSA task under uncertain missing modalities. Specifically, we propose a fine-grained representation factorization module that sufficiently extracts valuable sentiment information by factorizing modality into sentiment-relevant and modality-specific representations through crossmodal translation and sentiment semantic reconstruction. Moreover, a hierarchical mutual information maximization mechanism is introduced to incrementally maximize the mutual information between multi-scale representations to align and reconstruct the high-level semantics in the representations. Ultimately, we propose a hierarchical adversarial learning mechanism that further aligns and adapts the latent distribution of sentiment-relevant representations to produce robust joint multimodal representations. Comprehensive experiments on three datasets demonstrate that HRLF significantly improves MSA performance under uncertain modality missing cases. |
| title | Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.02793 |