Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.10518 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912974716796928 |
|---|---|
| author | Wang, Qunzhong Liu, Jie Liang, Jiajun Jiang, Yilei Zhang, Yuanxing Zheng, Yaozhi Wang, Xintao Wan, Pengfei Yue, Xiangyu Liu, Jiaheng |
| author_facet | Wang, Qunzhong Liu, Jie Liang, Jiajun Jiang, Yilei Zhang, Yuanxing Zheng, Yaozhi Wang, Xintao Wan, Pengfei Yue, Xiangyu Liu, Jiaheng |
| contents | Recent advancements in multimodal reward models (RMs) have substantially improved post-training for visual generative models. However, current RMs face inherent limitations: (1) visual inputs consume large context budgets, forcing fewer frames and causing loss of fine-grained details; and (2) all visual information is packed into the initial prompt, exacerbating hallucination and forgetting during chain-of-thought reasoning. To overcome these issues, we introduce VideoReward Thinker (VR-Thinker), a thinking-with-image framework that equips the RM with visual reasoning operations (e.g., select frame) and a configurable visual memory window. This allows the RM to actively acquire and update visual evidence within context limits, improving reasoning fidelity and reliability. We activate visual reasoning via a reinforcement fine-tuning pipeline: (i) Cold Start with curated visual chain-of-thought data to distill basic reasoning skills and operation formatting; (ii) select samples whose per-dimension and overall judgments are all correct, then conduct Rejection sampling Fine-Tuning on these high-quality traces to further enhance reasoning; and (iii) apply Group Relative Policy Optimization (GRPO) to strengthen reasoning. Our approach delivers state-of-the-art accuracy among open-source models on video preference benchmarks, especially for longer videos: a 7B VR-Thinker achieves 80.5% on VideoGen Reward, 82.3% on GenAI-Bench, and 75.6% on MJ-Bench-Video. These results validate the effectiveness and promise of thinking-with-image multimodal reward modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10518 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning Wang, Qunzhong Liu, Jie Liang, Jiajun Jiang, Yilei Zhang, Yuanxing Zheng, Yaozhi Wang, Xintao Wan, Pengfei Yue, Xiangyu Liu, Jiaheng Computer Vision and Pattern Recognition Recent advancements in multimodal reward models (RMs) have substantially improved post-training for visual generative models. However, current RMs face inherent limitations: (1) visual inputs consume large context budgets, forcing fewer frames and causing loss of fine-grained details; and (2) all visual information is packed into the initial prompt, exacerbating hallucination and forgetting during chain-of-thought reasoning. To overcome these issues, we introduce VideoReward Thinker (VR-Thinker), a thinking-with-image framework that equips the RM with visual reasoning operations (e.g., select frame) and a configurable visual memory window. This allows the RM to actively acquire and update visual evidence within context limits, improving reasoning fidelity and reliability. We activate visual reasoning via a reinforcement fine-tuning pipeline: (i) Cold Start with curated visual chain-of-thought data to distill basic reasoning skills and operation formatting; (ii) select samples whose per-dimension and overall judgments are all correct, then conduct Rejection sampling Fine-Tuning on these high-quality traces to further enhance reasoning; and (iii) apply Group Relative Policy Optimization (GRPO) to strengthen reasoning. Our approach delivers state-of-the-art accuracy among open-source models on video preference benchmarks, especially for longer videos: a 7B VR-Thinker achieves 80.5% on VideoGen Reward, 82.3% on GenAI-Bench, and 75.6% on MJ-Bench-Video. These results validate the effectiveness and promise of thinking-with-image multimodal reward modeling. |
| title | VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.10518 |