Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.13156 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910846961057792 |
|---|---|
| author | Liu, Tianqi Xiong, Wei Ren, Jie Chen, Lichang Wu, Junru Joshi, Rishabh Gao, Yang Shen, Jiaming Qin, Zhen Yu, Tianhe Sohn, Daniel Makarova, Anastasiia Liu, Jeremiah Liu, Yuan Piot, Bilal Ittycheriah, Abe Kumar, Aviral Saleh, Mohammad |
| author_facet | Liu, Tianqi Xiong, Wei Ren, Jie Chen, Lichang Wu, Junru Joshi, Rishabh Gao, Yang Shen, Jiaming Qin, Zhen Yu, Tianhe Sohn, Daniel Makarova, Anastasiia Liu, Jeremiah Liu, Yuan Piot, Bilal Ittycheriah, Abe Kumar, Aviral Saleh, Mohammad |
| contents | Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them. Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_13156 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RRM: Robust Reward Model Training Mitigates Reward Hacking Liu, Tianqi Xiong, Wei Ren, Jie Chen, Lichang Wu, Junru Joshi, Rishabh Gao, Yang Shen, Jiaming Qin, Zhen Yu, Tianhe Sohn, Daniel Makarova, Anastasiia Liu, Jeremiah Liu, Yuan Piot, Bilal Ittycheriah, Abe Kumar, Aviral Saleh, Mohammad Computation and Language Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them. Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%. |
| title | RRM: Robust Reward Model Training Mitigates Reward Hacking |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.13156 |