Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912877486538752 |
|---|---|
| author | Wang, Binghai Liu, Yantao Liu, Yuxuan Tang, Tianyi Wang, Shenzhi Gao, Chang Zheng, Chujie Zhang, Yichang Yu, Le Liu, Shixuan Gui, Tao Zhang, Qi Huang, Xuanjing Yu, Bowen Huang, Fei Lin, Junyang |
| author_facet | Wang, Binghai Liu, Yantao Liu, Yuxuan Tang, Tianyi Wang, Shenzhi Gao, Chang Zheng, Chujie Zhang, Yichang Yu, Le Liu, Shixuan Gui, Tao Zhang, Qi Huang, Xuanjing Yu, Bowen Huang, Fei Lin, Junyang |
| contents | Generative Reward Models (GenRMs) and LLM-as-a-Judge exhibit deceptive alignment by producing correct judgments for incorrect reasons, as they are trained and evaluated to prioritize Outcome Accuracy, which undermines their ability to generalize during RLHF. We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment. Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models and detects deceptive alignment, while outcome accuracy falls short in both respects. To mitigate this gap, we introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training. Our training method achieves state-of-the-art performance on RM-Bench (87.1%) and JudgeBench (82%), surpassing outcome-only baselines by an average of 5%. Using RM during RLHF, our method effectively improves performance as demonstrated on Arena Hard v2, notably yielding a 7% improvement in creative writing tasks. Further analysis confirms that our method escapes the deceptive alignment trap, effectively reversing the decline in rationale consistency observed in outcome-only training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_04649 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models Wang, Binghai Liu, Yantao Liu, Yuxuan Tang, Tianyi Wang, Shenzhi Gao, Chang Zheng, Chujie Zhang, Yichang Yu, Le Liu, Shixuan Gui, Tao Zhang, Qi Huang, Xuanjing Yu, Bowen Huang, Fei Lin, Junyang Computation and Language Generative Reward Models (GenRMs) and LLM-as-a-Judge exhibit deceptive alignment by producing correct judgments for incorrect reasons, as they are trained and evaluated to prioritize Outcome Accuracy, which undermines their ability to generalize during RLHF. We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment. Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models and detects deceptive alignment, while outcome accuracy falls short in both respects. To mitigate this gap, we introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training. Our training method achieves state-of-the-art performance on RM-Bench (87.1%) and JudgeBench (82%), surpassing outcome-only baselines by an average of 5%. Using RM during RLHF, our method effectively improves performance as demonstrated on Arena Hard v2, notably yielding a 7% improvement in creative writing tasks. Further analysis confirms that our method escapes the deceptive alignment trap, effectively reversing the decline in rationale consistency observed in outcome-only training. |
| title | Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.04649 |