Towards Multimodal Sentiment Analysis Debiasing via Bias Purification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916313080791040 |
|---|---|
| author | Yang, Dingkang Li, Mingcheng Xiao, Dongling Liu, Yang Yang, Kun Chen, Zhaoyu Wang, Yuzheng Zhai, Peng Li, Ke Zhang, Lihua |
| author_facet | Yang, Dingkang Li, Mingcheng Xiao, Dongling Liu, Yang Yang, Kun Chen, Zhaoyu Wang, Yuzheng Zhai, Peng Li, Ke Zhang, Lihua |
| contents | Multimodal Sentiment Analysis (MSA) aims to understand human intentions by integrating emotion-related clues from diverse modalities, such as visual, language, and audio. Unfortunately, the current MSA task invariably suffers from unplanned dataset biases, particularly multimodal utterance-level label bias and word-level context bias. These harmful biases potentially mislead models to focus on statistical shortcuts and spurious correlations, causing severe performance bottlenecks. To alleviate these issues, we present a Multimodal Counterfactual Inference Sentiment (MCIS) analysis framework based on causality rather than conventional likelihood. Concretely, we first formulate a causal graph to discover harmful biases from already-trained vanilla models. In the inference phase, given a factual multimodal input, MCIS imagines two counterfactual scenarios to purify and mitigate these biases. Then, MCIS can make unbiased decisions from biased observations by comparing factual and counterfactual outcomes. We conduct extensive experiments on several standard MSA benchmarks. Qualitative and quantitative results show the effectiveness of the proposed framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_05023 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Multimodal Sentiment Analysis Debiasing via Bias Purification Yang, Dingkang Li, Mingcheng Xiao, Dongling Liu, Yang Yang, Kun Chen, Zhaoyu Wang, Yuzheng Zhai, Peng Li, Ke Zhang, Lihua Computation and Language Computer Vision and Pattern Recognition Multimodal Sentiment Analysis (MSA) aims to understand human intentions by integrating emotion-related clues from diverse modalities, such as visual, language, and audio. Unfortunately, the current MSA task invariably suffers from unplanned dataset biases, particularly multimodal utterance-level label bias and word-level context bias. These harmful biases potentially mislead models to focus on statistical shortcuts and spurious correlations, causing severe performance bottlenecks. To alleviate these issues, we present a Multimodal Counterfactual Inference Sentiment (MCIS) analysis framework based on causality rather than conventional likelihood. Concretely, we first formulate a causal graph to discover harmful biases from already-trained vanilla models. In the inference phase, given a factual multimodal input, MCIS imagines two counterfactual scenarios to purify and mitigate these biases. Then, MCIS can make unbiased decisions from biased observations by comparing factual and counterfactual outcomes. We conduct extensive experiments on several standard MSA benchmarks. Qualitative and quantitative results show the effectiveness of the proposed framework. |
| title | Towards Multimodal Sentiment Analysis Debiasing via Bias Purification |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.05023 |