Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.06763 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918325913649152 |
|---|---|
| author | Lai, Yanlin Huang, Mitt Guo, Hangyu Wang, Xiangfeng Li, Haodong Zhan, Shaoxiong Zhao, Liang Yao, Chengyuan Zhang, Yinmin Han, Qi Yuan, Chun Ge, Zheng Zhang, Xiangyu Jiang, Daxin |
| author_facet | Lai, Yanlin Huang, Mitt Guo, Hangyu Wang, Xiangfeng Li, Haodong Zhan, Shaoxiong Zhao, Liang Yao, Chengyuan Zhang, Yinmin Han, Qi Yuan, Chun Ge, Zheng Zhang, Xiangyu Jiang, Daxin |
| contents | Reinforcement Learning from Human Feedback (RLHF) remains indispensable for aligning large language models (LLMs) in subjective domains. To enhance robustness, recent work shifts toward Generative Reward Models (GenRMs) that generate rationales before predicting preferences. Yet in GenRM training and evaluation, practice remains outcome-label-only, leaving reasoning quality unchecked. We show that reasoning fidelity-the consistency between a GenRM's preference decision and reference decision rationales-is highly predictive of downstream RLHF outcomes, beyond standard label accuracy. Specifically, we repurpose existing reward-model benchmarks to compute Spurious Correctness (S-Corr)-the fraction of label-correct decisions with rationales misaligned with golden judgments. Our empirical evaluation reveals substantial S-Corr even for competitive GenRMs, and higher S-Corr is associated with policy degeneration under optimization. To improve fidelity, we propose Rationale-Centric Alignment, R-Align, which augments training with gold judgments and explicitly supervises rationale alignment. R-Align reduces S-Corr on RM benchmarks and yields consistent gains in actor performance across STEM, coding, instruction following, and general tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06763 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging Lai, Yanlin Huang, Mitt Guo, Hangyu Wang, Xiangfeng Li, Haodong Zhan, Shaoxiong Zhao, Liang Yao, Chengyuan Zhang, Yinmin Han, Qi Yuan, Chun Ge, Zheng Zhang, Xiangyu Jiang, Daxin Computation and Language Reinforcement Learning from Human Feedback (RLHF) remains indispensable for aligning large language models (LLMs) in subjective domains. To enhance robustness, recent work shifts toward Generative Reward Models (GenRMs) that generate rationales before predicting preferences. Yet in GenRM training and evaluation, practice remains outcome-label-only, leaving reasoning quality unchecked. We show that reasoning fidelity-the consistency between a GenRM's preference decision and reference decision rationales-is highly predictive of downstream RLHF outcomes, beyond standard label accuracy. Specifically, we repurpose existing reward-model benchmarks to compute Spurious Correctness (S-Corr)-the fraction of label-correct decisions with rationales misaligned with golden judgments. Our empirical evaluation reveals substantial S-Corr even for competitive GenRMs, and higher S-Corr is associated with policy degeneration under optimization. To improve fidelity, we propose Rationale-Centric Alignment, R-Align, which augments training with gold judgments and explicitly supervises rationale alignment. R-Align reduces S-Corr on RM benchmarks and yields consistent gains in actor performance across STEM, coding, instruction following, and general tasks. |
| title | R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.06763 |