Contrastive Knowledge Distillation for Robust Multimodal Sentiment Analysis
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912069172854784 |
|---|---|
| author | Sang, Zhongyi Funakoshi, Kotaro Okumura, Manabu |
| author_facet | Sang, Zhongyi Funakoshi, Kotaro Okumura, Manabu |
| contents | Multimodal sentiment analysis (MSA) systems leverage information from different modalities to predict human sentiment intensities. Incomplete modality is an important issue that may cause a significant performance drop in MSA systems. By generative imputation, i.e., recovering the missing data from available data, systems may achieve robust performance but will lead to high computational costs. This paper introduces a knowledge distillation method, called `Multi-Modal Contrastive Knowledge Distillation' (MM-CKD), to address the issue of incomplete modality in video sentiment analysis with lower computation cost, as a novel non-imputation-based method. We employ Multi-view Supervised Contrastive Learning (MVSC) to transfer knowledge from a teacher model to student models. This approach not only leverages cross-modal knowledge but also introduces cross-sample knowledge with supervision, jointly improving the performance of both teacher and student models through online learning. Our method gives competitive results with significantly lower computational costs than state-of-the-art imputation-based methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08692 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Contrastive Knowledge Distillation for Robust Multimodal Sentiment Analysis Sang, Zhongyi Funakoshi, Kotaro Okumura, Manabu Multimedia Multimodal sentiment analysis (MSA) systems leverage information from different modalities to predict human sentiment intensities. Incomplete modality is an important issue that may cause a significant performance drop in MSA systems. By generative imputation, i.e., recovering the missing data from available data, systems may achieve robust performance but will lead to high computational costs. This paper introduces a knowledge distillation method, called `Multi-Modal Contrastive Knowledge Distillation' (MM-CKD), to address the issue of incomplete modality in video sentiment analysis with lower computation cost, as a novel non-imputation-based method. We employ Multi-view Supervised Contrastive Learning (MVSC) to transfer knowledge from a teacher model to student models. This approach not only leverages cross-modal knowledge but also introduces cross-sample knowledge with supervision, jointly improving the performance of both teacher and student models through online learning. Our method gives competitive results with significantly lower computational costs than state-of-the-art imputation-based methods. |
| title | Contrastive Knowledge Distillation for Robust Multimodal Sentiment Analysis |
| topic | Multimedia |
| url | https://arxiv.org/abs/2410.08692 |