Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.17291 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916020412743680 |
|---|---|
| author | Xie, Weichu Zhao, Haozhe Liu, Wenpu Zhu, Yongfu Chen, Liang Ye, Minghao Chen, Zirong Xu, Yuqi Dong, Shuai Wang, Ziyue Xu, Xinbo Shi, Kean Wu, Ruoyu Zhang, Xiaoying Shao, Wenqi Chang, Baobao Duan, Nan Wang, Jiaqi |
| author_facet | Xie, Weichu Zhao, Haozhe Liu, Wenpu Zhu, Yongfu Chen, Liang Ye, Minghao Chen, Zirong Xu, Yuqi Dong, Shuai Wang, Ziyue Xu, Xinbo Shi, Kean Wu, Ruoyu Zhang, Xiaoying Shao, Wenqi Chang, Baobao Duan, Nan Wang, Jiaqi |
| contents | Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models, but rewards only final-answer correctness with no supervision over intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) introduce finer-grained supervision by scoring rollouts against structured criteria, yet the rubric scores are still aggregated into a single scalar applied to the entire response, causing three weaknesses: loss of multi-criterion structure, uniform supervision of correct and incorrect steps, and reward hacking through unbounded self-correction. On 1,000 problems, we find 18.2% of steps in correct-answer responses are wrong yet positively rewarded, while 49.9% of steps in incorrect-answer responses are correct yet penalized. We introduce Step-wise Rubrics as Rewards (SRaR), an RLVR framework that (i) uses an LLM judge to attribute each rubric item to a specific reasoning step, (ii) normalizes per-step rubric scores across rollouts so only steps whose quality varies produce a learning signal, and (iii) combines the per-step reward with the outcome reward through a decoupled advantage estimator that keeps the outcome baseline stable. We further build a 16K-problem rubric dataset by contrastively distilling rubric items from correct and flawed reasoning paths sampled from a strong model. Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B, raises the Faithful Reasoning Rate on AIME 2025 from 34.5% to 46.7%, and reduces self-correction looping from 48.1% to 26.5%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_17291 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Step-wise Rubric Rewards for LLM Reasoning Xie, Weichu Zhao, Haozhe Liu, Wenpu Zhu, Yongfu Chen, Liang Ye, Minghao Chen, Zirong Xu, Yuqi Dong, Shuai Wang, Ziyue Xu, Xinbo Shi, Kean Wu, Ruoyu Zhang, Xiaoying Shao, Wenqi Chang, Baobao Duan, Nan Wang, Jiaqi Machine Learning Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models, but rewards only final-answer correctness with no supervision over intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) introduce finer-grained supervision by scoring rollouts against structured criteria, yet the rubric scores are still aggregated into a single scalar applied to the entire response, causing three weaknesses: loss of multi-criterion structure, uniform supervision of correct and incorrect steps, and reward hacking through unbounded self-correction. On 1,000 problems, we find 18.2% of steps in correct-answer responses are wrong yet positively rewarded, while 49.9% of steps in incorrect-answer responses are correct yet penalized. We introduce Step-wise Rubrics as Rewards (SRaR), an RLVR framework that (i) uses an LLM judge to attribute each rubric item to a specific reasoning step, (ii) normalizes per-step rubric scores across rollouts so only steps whose quality varies produce a learning signal, and (iii) combines the per-step reward with the outcome reward through a decoupled advantage estimator that keeps the outcome baseline stable. We further build a 16K-problem rubric dataset by contrastively distilling rubric items from correct and flawed reasoning paths sampled from a strong model. Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points on Qwen3-8B and 2.75 points on Qwen3-32B, raises the Faithful Reasoning Rate on AIME 2025 from 34.5% to 46.7%, and reduces self-correction looping from 48.1% to 26.5%. |
| title | Step-wise Rubric Rewards for LLM Reasoning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.17291 |