Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912312501207040 |
|---|---|
| author | Wang, Weiyun Chen, Zhe Wang, Wenhai Cao, Yue Liu, Yangzhou Gao, Zhangwei Zhu, Jinguo Zhu, Xizhou Lu, Lewei Qiao, Yu Dai, Jifeng |
| author_facet | Wang, Weiyun Chen, Zhe Wang, Wenhai Cao, Yue Liu, Yangzhou Gao, Zhangwei Zhu, Jinguo Zhu, Xizhou Lu, Lewei Qiao, Yu Dai, Jifeng |
| contents | Existing open-source multimodal large language models (MLLMs) generally follow a training process involving pre-training and supervised fine-tuning. However, these models suffer from distribution shifts, which limit their multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance. To address this, we introduce a preference optimization (PO) process to enhance the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data side, we design an automated preference data construction pipeline to create MMPR, a high-quality, large-scale multimodal reasoning preference dataset; and (2) on the model side, we explore integrating PO with MLLMs, developing a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance. Our approach enhances the multimodal reasoning abilities of both InternVL2-8B and InternVL2-76B. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance comparable to the 10$\times$ larger InternVL2-76B. We hope this study could inspire further advancements in MLLMs. Code, data, and model are released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10442 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Wang, Weiyun Chen, Zhe Wang, Wenhai Cao, Yue Liu, Yangzhou Gao, Zhangwei Zhu, Jinguo Zhu, Xizhou Lu, Lewei Qiao, Yu Dai, Jifeng Computation and Language Computer Vision and Pattern Recognition Existing open-source multimodal large language models (MLLMs) generally follow a training process involving pre-training and supervised fine-tuning. However, these models suffer from distribution shifts, which limit their multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance. To address this, we introduce a preference optimization (PO) process to enhance the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data side, we design an automated preference data construction pipeline to create MMPR, a high-quality, large-scale multimodal reasoning preference dataset; and (2) on the model side, we explore integrating PO with MLLMs, developing a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance. Our approach enhances the multimodal reasoning abilities of both InternVL2-8B and InternVL2-76B. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance comparable to the 10$\times$ larger InternVL2-76B. We hope this study could inspire further advancements in MLLMs. Code, data, and model are released. |
| title | Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.10442 |