Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.07263 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908398492057600 |
|---|---|
| author | Wang, Xiaokun Wang, Peiyu Pei, Jiangbo Shen, Wei Peng, Yi Hao, Yunzhuo Qiu, Weijie Jian, Ai Xie, Tianyidan Song, Xuchen Liu, Yang Zhou, Yahui |
| author_facet | Wang, Xiaokun Wang, Peiyu Pei, Jiangbo Shen, Wei Peng, Yi Hao, Yunzhuo Qiu, Weijie Jian, Ai Xie, Tianyidan Song, Xuchen Liu, Yang Zhou, Yahui |
| contents | We propose Skywork-VL Reward, a multimodal reward model that provides reward signals for both multimodal understanding and reasoning tasks. Our technical approach comprises two key components: First, we construct a large-scale multimodal preference dataset that covers a wide range of tasks and scenarios, with responses collected from both standard vision-language models (VLMs) and advanced VLM reasoners. Second, we design a reward model architecture based on Qwen2.5-VL-7B-Instruct, integrating a reward head and applying multi-stage fine-tuning using pairwise ranking loss on pairwise preference data. Experimental evaluations show that Skywork-VL Reward achieves state-of-the-art results on multimodal VL-RewardBench and exhibits competitive performance on the text-only RewardBench benchmark. Furthermore, preference data constructed based on our Skywork-VL Reward proves highly effective for training Mixed Preference Optimization (MPO), leading to significant improvements in multimodal reasoning capabilities. Our results underscore Skywork-VL Reward as a significant advancement toward general-purpose, reliable reward models for multimodal alignment. Our model has been publicly released to promote transparency and reproducibility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_07263 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning Wang, Xiaokun Wang, Peiyu Pei, Jiangbo Shen, Wei Peng, Yi Hao, Yunzhuo Qiu, Weijie Jian, Ai Xie, Tianyidan Song, Xuchen Liu, Yang Zhou, Yahui Computer Vision and Pattern Recognition We propose Skywork-VL Reward, a multimodal reward model that provides reward signals for both multimodal understanding and reasoning tasks. Our technical approach comprises two key components: First, we construct a large-scale multimodal preference dataset that covers a wide range of tasks and scenarios, with responses collected from both standard vision-language models (VLMs) and advanced VLM reasoners. Second, we design a reward model architecture based on Qwen2.5-VL-7B-Instruct, integrating a reward head and applying multi-stage fine-tuning using pairwise ranking loss on pairwise preference data. Experimental evaluations show that Skywork-VL Reward achieves state-of-the-art results on multimodal VL-RewardBench and exhibits competitive performance on the text-only RewardBench benchmark. Furthermore, preference data constructed based on our Skywork-VL Reward proves highly effective for training Mixed Preference Optimization (MPO), leading to significant improvements in multimodal reasoning capabilities. Our results underscore Skywork-VL Reward as a significant advancement toward general-purpose, reliable reward models for multimodal alignment. Our model has been publicly released to promote transparency and reproducibility. |
| title | Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.07263 |