Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914486905995264 |
|---|---|
| author | Liu, Weide Zhan, Huijing |
| author_facet | Liu, Weide Zhan, Huijing |
| contents | Multimodal sentiment analysis aims to identify the emotions expressed by individuals through visual, language, and acoustic cues. However, most existing research assume that all modalities are available during both training and testing, which makes their algorithms susceptible to the missing-modality scenarios. In this paper, we propose a novel knowledge-transfer network to translate between different modalities to reconstruct the missing audio features. Moreover, we develop a cross-modality attention mechanism to maximize the information extracted from the reconstructed and observed modalities for sentiment prediction. Extensive experiments on three publicly available datasets demonstrate significant improvements over baseline methods and achieve comparable results to the previous methods with complete multi-modality supervision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_10747 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach Liu, Weide Zhan, Huijing Sound Artificial Intelligence Computation and Language Machine Learning Audio and Speech Processing Multimodal sentiment analysis aims to identify the emotions expressed by individuals through visual, language, and acoustic cues. However, most existing research assume that all modalities are available during both training and testing, which makes their algorithms susceptible to the missing-modality scenarios. In this paper, we propose a novel knowledge-transfer network to translate between different modalities to reconstruct the missing audio features. Moreover, we develop a cross-modality attention mechanism to maximize the information extracted from the reconstructed and observed modalities for sentiment prediction. Extensive experiments on three publicly available datasets demonstrate significant improvements over baseline methods and achieve comparable results to the previous methods with complete multi-modality supervision. |
| title | Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach |
| topic | Sound Artificial Intelligence Computation and Language Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2401.10747 |