Real-Time Aligned Reward Model beyond Semantics
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913135433089024 |
|---|---|
| author | Huang, Zixuan Xia, Xin Ren, Yuxi Zheng, Jianbin Xiao, Xuefeng Xie, Hongyan Huaqiu, Li Liang, Songshi Dai, Zhongxiang Zhuang, Fuzhen Li, Jianxin Ban, Yikun Wang, Deqing |
| author_facet | Huang, Zixuan Xia, Xin Ren, Yuxi Zheng, Jianbin Xiao, Xuefeng Xie, Hongyan Huaqiu, Li Liang, Songshi Dai, Zhongxiang Zhuang, Fuzhen Li, Jianxin Ban, Yikun Wang, Deqing |
| contents | Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique for aligning large language models (LLMs) with human preferences, yet it is susceptible to reward overoptimization, in which policy models overfit to the reward model, exploit spurious reward patterns instead of faithfully capturing human intent. Prior mitigations primarily relies on surface semantic information and fails to efficiently address the misalignment between the reward model (RM) and the policy model caused by continuous policy distribution shifts. This inevitably leads to an increasing reward discrepancy, exacerbating reward overoptimization. To address these limitations, we introduce R2M (Real-Time Aligned Reward Model), a novel lightweight RLHF framework. R2M goes beyond vanilla reward models that solely depend on the semantic representations of a pretrained LLM. Instead, it leverages the evolving hidden states of the policy (namely policy feedback) to align with the real-time distribution shift of the policy during the RL process. This work points to a promising new direction for improving the performance of reward models through real-time utilization of feedback from policy models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22664 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Real-Time Aligned Reward Model beyond Semantics Huang, Zixuan Xia, Xin Ren, Yuxi Zheng, Jianbin Xiao, Xuefeng Xie, Hongyan Huaqiu, Li Liang, Songshi Dai, Zhongxiang Zhuang, Fuzhen Li, Jianxin Ban, Yikun Wang, Deqing Artificial Intelligence Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique for aligning large language models (LLMs) with human preferences, yet it is susceptible to reward overoptimization, in which policy models overfit to the reward model, exploit spurious reward patterns instead of faithfully capturing human intent. Prior mitigations primarily relies on surface semantic information and fails to efficiently address the misalignment between the reward model (RM) and the policy model caused by continuous policy distribution shifts. This inevitably leads to an increasing reward discrepancy, exacerbating reward overoptimization. To address these limitations, we introduce R2M (Real-Time Aligned Reward Model), a novel lightweight RLHF framework. R2M goes beyond vanilla reward models that solely depend on the semantic representations of a pretrained LLM. Instead, it leverages the evolving hidden states of the policy (namely policy feedback) to align with the real-time distribution shift of the policy during the RL process. This work points to a promising new direction for improving the performance of reward models through real-time utilization of feedback from policy models. |
| title | Real-Time Aligned Reward Model beyond Semantics |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.22664 |