Adversarial Training of Reward Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917982827970560 |
|---|---|
| author | Bukharin, Alexander Qian, Haifeng Sun, Shengyang Renduchintala, Adithya Singhal, Soumye Wang, Zhilin Kuchaiev, Oleksii Delalleau, Olivier Zhao, Tuo |
| author_facet | Bukharin, Alexander Qian, Haifeng Sun, Shengyang Renduchintala, Adithya Singhal, Soumye Wang, Zhilin Kuchaiev, Oleksii Delalleau, Olivier Zhao, Tuo |
| contents | Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples -- responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv-RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv-RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_06141 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adversarial Training of Reward Models Bukharin, Alexander Qian, Haifeng Sun, Shengyang Renduchintala, Adithya Singhal, Soumye Wang, Zhilin Kuchaiev, Oleksii Delalleau, Olivier Zhao, Tuo Machine Learning Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples -- responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv-RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv-RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings. |
| title | Adversarial Training of Reward Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.06141 |