SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916989506682880 |
|---|---|
| author | Wan, Zhongwei Dou, Zhihao Liu, Che Zhang, Yu Cui, Dongfei Zhao, Qinjian Shen, Hui Xiong, Jing Xin, Yi Jiang, Yifan Tao, Chaofan He, Yangfan Zhang, Mi Yan, Shen |
| author_facet | Wan, Zhongwei Dou, Zhihao Liu, Che Zhang, Yu Cui, Dongfei Zhao, Qinjian Shen, Hui Xiong, Jing Xin, Yi Jiang, Yifan Tao, Chaofan He, Yangfan Zhang, Mi Yan, Shen |
| contents | Multimodal large language models (MLLMs) have shown promising capabilities in reasoning tasks, yet still struggle with complex problems requiring explicit self-reflection and self-correction, especially compared to their unimodal text-based counterparts. Existing reflection methods are simplistic and struggle to generate meaningful and instructive feedback, as the reasoning ability and knowledge limits of pre-trained models are largely fixed during initial training. To overcome these challenges, we propose Multimodal Self-Reflection enhanced reasoning with Group Relative Policy Optimization (SRPO), a two-stage reflection-aware reinforcement learning (RL) framework explicitly designed to enhance multimodal LLM reasoning. In the first stage, we construct a high-quality, reflection-focused dataset under the guidance of an advanced MLLM, which generates reflections based on initial responses to help the policy model learn both reasoning and self-reflection. In the second stage, we introduce a novel reward mechanism within the GRPO framework that encourages concise and cognitively meaningful reflection while avoiding redundancy. Extensive experiments across multiple multimodal reasoning benchmarks, including MathVista, MathVision, MathVerse, and MMMU-Pro, using Qwen-2.5-VL-7B and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms state-of-the-art models, achieving notable improvements in both reasoning accuracy and reflection quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01713 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning Wan, Zhongwei Dou, Zhihao Liu, Che Zhang, Yu Cui, Dongfei Zhao, Qinjian Shen, Hui Xiong, Jing Xin, Yi Jiang, Yifan Tao, Chaofan He, Yangfan Zhang, Mi Yan, Shen Computation and Language Multimodal large language models (MLLMs) have shown promising capabilities in reasoning tasks, yet still struggle with complex problems requiring explicit self-reflection and self-correction, especially compared to their unimodal text-based counterparts. Existing reflection methods are simplistic and struggle to generate meaningful and instructive feedback, as the reasoning ability and knowledge limits of pre-trained models are largely fixed during initial training. To overcome these challenges, we propose Multimodal Self-Reflection enhanced reasoning with Group Relative Policy Optimization (SRPO), a two-stage reflection-aware reinforcement learning (RL) framework explicitly designed to enhance multimodal LLM reasoning. In the first stage, we construct a high-quality, reflection-focused dataset under the guidance of an advanced MLLM, which generates reflections based on initial responses to help the policy model learn both reasoning and self-reflection. In the second stage, we introduce a novel reward mechanism within the GRPO framework that encourages concise and cognitively meaningful reflection while avoiding redundancy. Extensive experiments across multiple multimodal reasoning benchmarks, including MathVista, MathVision, MathVerse, and MMMU-Pro, using Qwen-2.5-VL-7B and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms state-of-the-art models, achieving notable improvements in both reasoning accuracy and reflection quality. |
| title | SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.01713 |