Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912060851355648 |
|---|---|
| author | Feng, Xinyu Lin, Yuming He, Lihua Li, You Chang, Liang Zhou, Ya |
| author_facet | Feng, Xinyu Lin, Yuming He, Lihua Li, You Chang, Liang Zhou, Ya |
| contents | Multimodal Sentiment Analysis (MSA) utilizes multimodal data to infer the users' sentiment. Previous methods focus on equally treating the contribution of each modality or statically using text as the dominant modality to conduct interaction, which neglects the situation where each modality may become dominant. In this paper, we propose a Knowledge-Guided Dynamic Modality Attention Fusion Framework (KuDA) for multimodal sentiment analysis. KuDA uses sentiment knowledge to guide the model dynamically selecting the dominant modality and adjusting the contributions of each modality. In addition, with the obtained multimodal representation, the model can further highlight the contribution of dominant modality through the correlation evaluation loss. Extensive experiments on four MSA benchmark datasets indicate that KuDA achieves state-of-the-art performance and is able to adapt to different scenarios of dominant modality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04491 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis Feng, Xinyu Lin, Yuming He, Lihua Li, You Chang, Liang Zhou, Ya Computation and Language Artificial Intelligence Multimedia Multimodal Sentiment Analysis (MSA) utilizes multimodal data to infer the users' sentiment. Previous methods focus on equally treating the contribution of each modality or statically using text as the dominant modality to conduct interaction, which neglects the situation where each modality may become dominant. In this paper, we propose a Knowledge-Guided Dynamic Modality Attention Fusion Framework (KuDA) for multimodal sentiment analysis. KuDA uses sentiment knowledge to guide the model dynamically selecting the dominant modality and adjusting the contributions of each modality. In addition, with the obtained multimodal representation, the model can further highlight the contribution of dominant modality through the correlation evaluation loss. Extensive experiments on four MSA benchmark datasets indicate that KuDA achieves state-of-the-art performance and is able to adapt to different scenarios of dominant modality. |
| title | Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis |
| topic | Computation and Language Artificial Intelligence Multimedia |
| url | https://arxiv.org/abs/2410.04491 |