ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914488140169216 |
|---|---|
| author | Qin, Kai Liu, Liangxin Liang, Yu Wang, Longzheng Wang, Yan Zhang, Yueyang Xia, Long Sun, Zhiyuan Liu, Houde Shi, Daiting |
| author_facet | Qin, Kai Liu, Liangxin Liang, Yu Wang, Longzheng Wang, Yan Zhang, Yueyang Xia, Long Sun, Zhiyuan Liu, Houde Shi, Daiting |
| contents | Reward Models (RMs) are critical components in the Reinforcement Learning from Human Feedback (RLHF) pipeline, directly determining the alignment quality of Large Language Models (LLMs). Recently, Generative Reward Models (GRMs) have emerged as a superior paradigm, offering higher interpretability and stronger generalization than traditional scalar RMs. However, existing methods for GRMs focus primarily on outcome-level supervision, neglecting analytical process quality, which constrains their potential. To address this, we propose ReflectRM, a novel GRM that leverages self-reflection to assess analytical quality and enhance preference modeling. ReflectRM is trained under a unified generative framework for joint modeling of response preference and analysis preference. During inference, we use its self-reflection capability to identify the most reliable analysis, from which the final preference prediction is derived. Experiments across four benchmarks show that ReflectRM consistently improves performance, achieving an average accuracy gain of +3.7 on Qwen3-4B. Further experiments confirm that response preference and analysis preference are mutually reinforcing. Notably, ReflectRM substantially mitigates positional bias, yielding +10.2 improvement compared with leading GRMs and establishing itself as a more stable evaluator. Our code is available at https://github.com/yuliangCarmelo/ReflectRM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07506 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework Qin, Kai Liu, Liangxin Liang, Yu Wang, Longzheng Wang, Yan Zhang, Yueyang Xia, Long Sun, Zhiyuan Liu, Houde Shi, Daiting Artificial Intelligence Computation and Language Reward Models (RMs) are critical components in the Reinforcement Learning from Human Feedback (RLHF) pipeline, directly determining the alignment quality of Large Language Models (LLMs). Recently, Generative Reward Models (GRMs) have emerged as a superior paradigm, offering higher interpretability and stronger generalization than traditional scalar RMs. However, existing methods for GRMs focus primarily on outcome-level supervision, neglecting analytical process quality, which constrains their potential. To address this, we propose ReflectRM, a novel GRM that leverages self-reflection to assess analytical quality and enhance preference modeling. ReflectRM is trained under a unified generative framework for joint modeling of response preference and analysis preference. During inference, we use its self-reflection capability to identify the most reliable analysis, from which the final preference prediction is derived. Experiments across four benchmarks show that ReflectRM consistently improves performance, achieving an average accuracy gain of +3.7 on Qwen3-4B. Further experiments confirm that response preference and analysis preference are mutually reinforcing. Notably, ReflectRM substantially mitigates positional bias, yielding +10.2 improvement compared with leading GRMs and establishing itself as a more stable evaluator. Our code is available at https://github.com/yuliangCarmelo/ReflectRM. |
| title | ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2604.07506 |