Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915211204624384 |
|---|---|
| author | Zhang, Zefeng Tang, Hengzhu Sheng, Jiawei Zhang, Zhenyu Ren, Yiming Li, Zhenyang Yin, Dawei Ma, Duohe Liu, Tingwen |
| author_facet | Zhang, Zefeng Tang, Hengzhu Sheng, Jiawei Zhang, Zhenyu Ren, Yiming Li, Zhenyang Yin, Dawei Ma, Duohe Liu, Tingwen |
| contents | Multimodal Large Language Models excel in various tasks, yet often struggle with modality bias, where the model tends to rely heavily on a single modality and overlook critical information in other modalities, which leads to incorrect focus and generating irrelevant responses. In this paper, we propose using the paradigm of preference optimization to solve the modality bias problem, including RLAIFVBias, a debiased preference optimization dataset, and a Noise Aware Preference Optimization algorithm. Specifically, we first construct the dataset by introducing perturbations to reduce the informational content of certain modalities, compelling the model to rely on a specific modality when generating negative responses. To address the inevitable noise in automatically constructed data, we combine the noise robust Mean Absolute Error with the Binary Cross Entropy in Direct Preference Optimization by a negative Box Cox transformation, and dynamically adjust the algorithm noise robustness based on the evaluated noise levels in the data. Extensive experiments validate our approach, demonstrating not only its effectiveness in mitigating modality bias but also its significant role in minimizing hallucinations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_17928 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization Zhang, Zefeng Tang, Hengzhu Sheng, Jiawei Zhang, Zhenyu Ren, Yiming Li, Zhenyang Yin, Dawei Ma, Duohe Liu, Tingwen Computer Vision and Pattern Recognition Computation and Language Multimodal Large Language Models excel in various tasks, yet often struggle with modality bias, where the model tends to rely heavily on a single modality and overlook critical information in other modalities, which leads to incorrect focus and generating irrelevant responses. In this paper, we propose using the paradigm of preference optimization to solve the modality bias problem, including RLAIFVBias, a debiased preference optimization dataset, and a Noise Aware Preference Optimization algorithm. Specifically, we first construct the dataset by introducing perturbations to reduce the informational content of certain modalities, compelling the model to rely on a specific modality when generating negative responses. To address the inevitable noise in automatically constructed data, we combine the noise robust Mean Absolute Error with the Binary Cross Entropy in Direct Preference Optimization by a negative Box Cox transformation, and dynamically adjust the algorithm noise robustness based on the evaluated noise levels in the data. Extensive experiments validate our approach, demonstrating not only its effectiveness in mitigating modality bias but also its significant role in minimizing hallucinations. |
| title | Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2503.17928 |