Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.23235 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911028182253568 |
|---|---|
| author | Li, Yi-Chen Xu, Tian Yu, Yang Zhang, Xuqin Chen, Xiong-Hui Ling, Zhongxiang Chao, Ningjing Yuan, Lei Zhou, Zhi-Hua |
| author_facet | Li, Yi-Chen Xu, Tian Yu, Yang Zhang, Xuqin Chen, Xiong-Hui Ling, Zhongxiang Chao, Ningjing Yuan, Lei Zhou, Zhi-Hua |
| contents | The alignment of Large Language Models (LLMs) is critically dependent on reward models trained on costly human preference data. While recent work explores bypassing this cost with AI feedback, these methods often lack a rigorous theoretical foundation. In this paper, we discover that a powerful generalist reward model is already latently present within any LLM trained via standard next-token prediction. We prove that this endogenous reward is not a heuristic, but is theoretically equivalent to a reward function learned through offline inverse reinforcement learning. This connection allows us to directly elicit a high-quality reward signal from a base (pre-trained or supervised fine-tuned) model without any further training. Critically, we also prove that subsequent reinforcement learning using this endogenous reward leads to a policy with a provably superior error bound compared to the base model. To our best knowledge, this is the first theoretical proof of the effectiveness of reinforcement learning for LLMs. Our experiments validate this theory, demonstrating that our method not only outperforms existing LLM-as-a-judge approaches but can also surpass explicitly trained reward models. These findings suggest that the reward modeling stage can be replaced by a principled method of eliciting the knowledge already captured during pre-training, heralding a more efficient, powerful, and scalable paradigm for LLMs alignment as well as multi-modal models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_23235 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Generalist Reward Models: Found Inside Large Language Models Li, Yi-Chen Xu, Tian Yu, Yang Zhang, Xuqin Chen, Xiong-Hui Ling, Zhongxiang Chao, Ningjing Yuan, Lei Zhou, Zhi-Hua Computation and Language The alignment of Large Language Models (LLMs) is critically dependent on reward models trained on costly human preference data. While recent work explores bypassing this cost with AI feedback, these methods often lack a rigorous theoretical foundation. In this paper, we discover that a powerful generalist reward model is already latently present within any LLM trained via standard next-token prediction. We prove that this endogenous reward is not a heuristic, but is theoretically equivalent to a reward function learned through offline inverse reinforcement learning. This connection allows us to directly elicit a high-quality reward signal from a base (pre-trained or supervised fine-tuned) model without any further training. Critically, we also prove that subsequent reinforcement learning using this endogenous reward leads to a policy with a provably superior error bound compared to the base model. To our best knowledge, this is the first theoretical proof of the effectiveness of reinforcement learning for LLMs. Our experiments validate this theory, demonstrating that our method not only outperforms existing LLM-as-a-judge approaches but can also surpass explicitly trained reward models. These findings suggest that the reward modeling stage can be replaced by a principled method of eliciting the knowledge already captured during pre-training, heralding a more efficient, powerful, and scalable paradigm for LLMs alignment as well as multi-modal models. |
| title | Generalist Reward Models: Found Inside Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.23235 |