Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915897362350080 |
|---|---|
| author | Feiding Zhang, Yongkang Liao, Yuhao Zeng, Zijian Zhu, Chunzheng Zheng, Yaozong Liu, Yafei Peng, Yeling Wang, Youwei Wang, Sibo Yang, Huiming Liao, Linglin Yang, Shunzhi |
| author_facet | Feiding Zhang, Yongkang Liao, Yuhao Zeng, Zijian Zhu, Chunzheng Zheng, Yaozong Liu, Yafei Peng, Yeling Wang, Youwei Wang, Sibo Yang, Huiming Liao, Linglin Yang, Shunzhi |
| contents | Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision--reasoning process alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_27482 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning Feiding Zhang, Yongkang Liao, Yuhao Zeng, Zijian Zhu, Chunzheng Zheng, Yaozong Liu, Yafei Peng, Yeling Wang, Youwei Wang, Sibo Yang, Huiming Liao, Linglin Yang, Shunzhi Computer Vision and Pattern Recognition Artificial Intelligence Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields sparse credit assignment in multi-step reasoning, weakening the linkage between visual evidence and intermediate steps and often causing unstable optimization and visual hallucinations. We propose Differential Feedback, which automatically constructs token/step-level supervision masks by repairing erroneous reasoning trajectories, explicitly marking the key positions that require correction. Without costly large-scale step-by-step human annotations, our method enables process-level visual alignment and can be seamlessly integrated into existing GRPO-like frameworks. Experiments on multimodal reasoning benchmarks including MMMStar and MathVista show an average 3% improvement under matched compute budgets. Our approach offers an effective, low-cost solution for accurate vision--reasoning process alignment. |
| title | Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2603.27482 |