Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909220121608192 |
|---|---|
| author | Li, Mingcheng Yang, Dingkang Zhao, Xiao Wang, Shuaibing Wang, Yan Yang, Kun Sun, Mingyang Kou, Dongliang Qian, Ziyun Zhang, Lihua |
| author_facet | Li, Mingcheng Yang, Dingkang Zhao, Xiao Wang, Shuaibing Wang, Yan Yang, Kun Sun, Mingyang Kou, Dongliang Qian, Ziyun Zhang, Lihua |
| contents | Multimodal sentiment analysis (MSA) aims to understand human sentiment through multimodal data. Most MSA efforts are based on the assumption of modality completeness. However, in real-world applications, some practical factors cause uncertain modality missingness, which drastically degrades the model's performance. To this end, we propose a Correlation-decoupled Knowledge Distillation (CorrKD) framework for the MSA task under uncertain missing modalities. Specifically, we present a sample-level contrastive distillation mechanism that transfers comprehensive knowledge containing cross-sample correlations to reconstruct missing semantics. Moreover, a category-guided prototype distillation mechanism is introduced to capture cross-category correlations using category prototypes to align feature distributions and generate favorable joint representations. Eventually, we design a response-disentangled consistency distillation strategy to optimize the sentiment decision boundaries of the student network through response disentanglement and mutual information maximization. Comprehensive experiments on three datasets indicate that our framework can achieve favorable improvements compared with several baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_16456 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities Li, Mingcheng Yang, Dingkang Zhao, Xiao Wang, Shuaibing Wang, Yan Yang, Kun Sun, Mingyang Kou, Dongliang Qian, Ziyun Zhang, Lihua Computer Vision and Pattern Recognition Multimodal sentiment analysis (MSA) aims to understand human sentiment through multimodal data. Most MSA efforts are based on the assumption of modality completeness. However, in real-world applications, some practical factors cause uncertain modality missingness, which drastically degrades the model's performance. To this end, we propose a Correlation-decoupled Knowledge Distillation (CorrKD) framework for the MSA task under uncertain missing modalities. Specifically, we present a sample-level contrastive distillation mechanism that transfers comprehensive knowledge containing cross-sample correlations to reconstruct missing semantics. Moreover, a category-guided prototype distillation mechanism is introduced to capture cross-category correlations using category prototypes to align feature distributions and generate favorable joint representations. Eventually, we design a response-disentangled consistency distillation strategy to optimize the sentiment decision boundaries of the student network through response disentanglement and mutual information maximization. Comprehensive experiments on three datasets indicate that our framework can achieve favorable improvements compared with several baselines. |
| title | Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.16456 |