Reinforcement Learning with Robust Rubric Rewards
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913170775343104 |
|---|---|
| author | Yu, Ya-Qi Wang, Hao Hong, Fangyu Qu, Xiangyang Wu, Gaojie Luo, Qiaoyu Xu, Nuo Wang, Huixin Xu, Wuheng Liao, Yongxin Chen, Zihao Li, Haonan Li, Ziming Peng, Dezhi Liao, Minghui Wu, Jihao Ren, Haoyu Tu, Dandan |
| author_facet | Yu, Ya-Qi Wang, Hao Hong, Fangyu Qu, Xiangyang Wu, Gaojie Luo, Qiaoyu Xu, Nuo Wang, Huixin Xu, Wuheng Liao, Yongxin Chen, Zihao Li, Haonan Li, Ziming Peng, Dezhi Liao, Minghui Wu, Jihao Ren, Haoyu Tu, Dandan |
| contents | While Reinforcement Learning with Verifiable Rewards (RLVR) is effective for deterministically checkable tasks, many vision-language tasks are partially verifiable, demanding multi-criteria supervision (e.g., perceptual details, reasoning steps, and constraints). Rubrics provide a natural interface for this fine-grained supervision, but their effectiveness depends on the execution accuracy during online RL. We propose Reinforcement Learning with Robust Rubric Rewards ($\text{RLR}^3$), extending RLVR from task-level verification to criterion-level verification. $\text{RLR}^3$ routes instance-specific rubrics through two execution paths: an LLM-as-an-extractor paired with a deterministic verifier, or an LLM-as-a-Judge for non-verifiable criteria. To ensure faithful scoring, $\text{RLR}^3$ introduce a minimal exposure strategy that masks ground truths from extractors and images from judges. Furthermore, $\text{RLR}^3$ employs hierarchical aggregation to prioritize essential criteria over additional criteria, and mitigates score saturation within rollout groups. Evaluated on Qwen3-VL-30B-A3B across 15 benchmarks, $\text{RLR}^3$ consistently outperforms RLVR, yielding a 4.7-point improvement over the base model and exceeding the official instruct-to-thinking model gap. Controlled audits confirm our deterministic verification and minimal exposure significantly reduce exploitable false positives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_30244 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reinforcement Learning with Robust Rubric Rewards Yu, Ya-Qi Wang, Hao Hong, Fangyu Qu, Xiangyang Wu, Gaojie Luo, Qiaoyu Xu, Nuo Wang, Huixin Xu, Wuheng Liao, Yongxin Chen, Zihao Li, Haonan Li, Ziming Peng, Dezhi Liao, Minghui Wu, Jihao Ren, Haoyu Tu, Dandan Computer Vision and Pattern Recognition Artificial Intelligence While Reinforcement Learning with Verifiable Rewards (RLVR) is effective for deterministically checkable tasks, many vision-language tasks are partially verifiable, demanding multi-criteria supervision (e.g., perceptual details, reasoning steps, and constraints). Rubrics provide a natural interface for this fine-grained supervision, but their effectiveness depends on the execution accuracy during online RL. We propose Reinforcement Learning with Robust Rubric Rewards ($\text{RLR}^3$), extending RLVR from task-level verification to criterion-level verification. $\text{RLR}^3$ routes instance-specific rubrics through two execution paths: an LLM-as-an-extractor paired with a deterministic verifier, or an LLM-as-a-Judge for non-verifiable criteria. To ensure faithful scoring, $\text{RLR}^3$ introduce a minimal exposure strategy that masks ground truths from extractors and images from judges. Furthermore, $\text{RLR}^3$ employs hierarchical aggregation to prioritize essential criteria over additional criteria, and mitigates score saturation within rollout groups. Evaluated on Qwen3-VL-30B-A3B across 15 benchmarks, $\text{RLR}^3$ consistently outperforms RLVR, yielding a 4.7-point improvement over the base model and exceeding the official instruct-to-thinking model gap. Controlled audits confirm our deterministic verification and minimal exposure significantly reduce exploitable false positives. |
| title | Reinforcement Learning with Robust Rubric Rewards |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2605.30244 |