Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.19616 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910003896516608 |
|---|---|
| author | Cai, Rui Li, Bangzheng Wen, Xiaofei Chen, Muhao Zhao, Zhe |
| author_facet | Cai, Rui Li, Bangzheng Wen, Xiaofei Chen, Muhao Zhao, Zhe |
| contents | Multimodal Large Language Models demonstrate strong performance on multimodal benchmarks, yet often exhibit poor robustness when exposed to spurious modality interference, such as irrelevant text in vision understanding, or irrelevant visual content in question answering. At its core, modality interference refers to cases where spurious signals from non-essential modalities distort model decisions, which we systematically analyze through causal, perturbation-based diagnostic experiments. To address this problem, we propose a unified finetuning framework that combines heuristic and adversarial perturbation-based data augmentation with output-level consistency regularization between original and perturbed inputs. Extensive experiments across image-heavy, text-heavy, and multimodal benchmarks, spanning multiple MLLM architectures and model scales, demonstrate consistent improvements in unimodal robustness and generalization, while improving standard multimodal performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19616 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models Cai, Rui Li, Bangzheng Wen, Xiaofei Chen, Muhao Zhao, Zhe Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Multimodal Large Language Models demonstrate strong performance on multimodal benchmarks, yet often exhibit poor robustness when exposed to spurious modality interference, such as irrelevant text in vision understanding, or irrelevant visual content in question answering. At its core, modality interference refers to cases where spurious signals from non-essential modalities distort model decisions, which we systematically analyze through causal, perturbation-based diagnostic experiments. To address this problem, we propose a unified finetuning framework that combines heuristic and adversarial perturbation-based data augmentation with output-level consistency regularization between original and perturbed inputs. Extensive experiments across image-heavy, text-heavy, and multimodal benchmarks, spanning multiple MLLM architectures and model scales, demonstrate consistent improvements in unimodal robustness and generalization, while improving standard multimodal performance. |
| title | Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.19616 |