Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.20369 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912665900679168 |
|---|---|
| author | Xu, Zhenghao Lu, Qin Zhang, Qingru Qiu, Liang Hong, Ilgee Yu, Changlong Yao, Wenlin Liu, Yao Jiang, Haoming Li, Lihong Yun, Hyokun Zhao, Tuo |
| author_facet | Xu, Zhenghao Lu, Qin Zhang, Qingru Qiu, Liang Hong, Ilgee Yu, Changlong Yao, Wenlin Liu, Yao Jiang, Haoming Li, Lihong Yun, Hyokun Zhao, Tuo |
| contents | Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to out-of-distribution (OOD) inputs. By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. In this work, we propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_20369 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ask a Strong LLM Judge when Your Reward Model is Uncertain Xu, Zhenghao Lu, Qin Zhang, Qingru Qiu, Liang Hong, Ilgee Yu, Changlong Yao, Wenlin Liu, Yao Jiang, Haoming Li, Lihong Yun, Hyokun Zhao, Tuo Machine Learning Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to out-of-distribution (OOD) inputs. By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. In this work, we propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF. |
| title | Ask a Strong LLM Judge when Your Reward Model is Uncertain |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.20369 |