Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2601.20327 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912865966882816 |
|---|---|
| author | Hu, Xinyu He, Yancheng Wang, Weixun Feng, Tao Lin, Li Liu, Jiashun Su, Wenbo Zheng, Bo Wan, Xiaojun |
| author_facet | Hu, Xinyu He, Yancheng Wang, Weixun Feng, Tao Lin, Li Liu, Jiashun Su, Wenbo Zheng, Bo Wan, Xiaojun |
| contents | Automatic evaluation is crucial yet challenging for open-ended natural language generation, especially when rule-based metrics are infeasible. Compared with traditional methods, the recent LLM-as-a-Judge paradigms enable better and more flexible evaluation, and show promise as generative reward models for reinforcement learning. However, prior work has revealed a notable gap between their seemingly impressive benchmark performance and actual effectiveness in RL practice. We attribute this issue to some limitations in existing studies, including the dominance of pairwise evaluation and inadequate optimization of evaluation criteria. Therefore, we propose CE-RM-4B, a pointwise generative reward model trained with a dedicated two-stage rollout method, and adopting unified query-based criteria. Using only about 5.7K high-quality data curated from the open-source preference dataset, our CE-RM-4B achieves superior performance on diverse reward model benchmarks, especially in Best-of-N scenarios, and delivers more effective improvements in downstream RL practice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_20327 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteria Hu, Xinyu He, Yancheng Wang, Weixun Feng, Tao Lin, Li Liu, Jiashun Su, Wenbo Zheng, Bo Wan, Xiaojun Computation and Language Automatic evaluation is crucial yet challenging for open-ended natural language generation, especially when rule-based metrics are infeasible. Compared with traditional methods, the recent LLM-as-a-Judge paradigms enable better and more flexible evaluation, and show promise as generative reward models for reinforcement learning. However, prior work has revealed a notable gap between their seemingly impressive benchmark performance and actual effectiveness in RL practice. We attribute this issue to some limitations in existing studies, including the dominance of pairwise evaluation and inadequate optimization of evaluation criteria. Therefore, we propose CE-RM-4B, a pointwise generative reward model trained with a dedicated two-stage rollout method, and adopting unified query-based criteria. Using only about 5.7K high-quality data curated from the open-source preference dataset, our CE-RM-4B achieves superior performance on diverse reward model benchmarks, especially in Best-of-N scenarios, and delivers more effective improvements in downstream RL practice. |
| title | CE-RM: A Pointwise Generative Reward Model Optimized via Two-Stage Rollout and Unified Criteria |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.20327 |