Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2401.12086 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912084706459648 |
|---|---|
| author | Pace, Alizée Mallinson, Jonathan Malmi, Eric Krause, Sebastian Severyn, Aliaksei |
| author_facet | Pace, Alizée Mallinson, Jonathan Malmi, Eric Krause, Sebastian Severyn, Aliaksei |
| contents | The success of reinforcement learning from human feedback (RLHF) in language model alignment is strongly dependent on the quality of the underlying reward model. In this paper, we present a novel approach to improve reward model quality by generating synthetic preference data, thereby augmenting the training dataset with on-policy, high-quality preference pairs. Motivated by the promising results of Best-of-N sampling strategies in language model training, we extend their application to reward model training. This results in a self-training strategy to generate preference pairs by selecting the best and worst candidates in a pool of responses to a given query. Empirically, we find that this approach improves the performance of any reward model, with an effect comparable to the addition of a similar quantity of human preference data. This work opens up new avenues of research for improving RLHF for language model alignment, by offering synthetic preference generation as a solution to reward modeling challenges. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_12086 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | West-of-N: Synthetic Preferences for Self-Improving Reward Models Pace, Alizée Mallinson, Jonathan Malmi, Eric Krause, Sebastian Severyn, Aliaksei Computation and Language Artificial Intelligence Machine Learning The success of reinforcement learning from human feedback (RLHF) in language model alignment is strongly dependent on the quality of the underlying reward model. In this paper, we present a novel approach to improve reward model quality by generating synthetic preference data, thereby augmenting the training dataset with on-policy, high-quality preference pairs. Motivated by the promising results of Best-of-N sampling strategies in language model training, we extend their application to reward model training. This results in a self-training strategy to generate preference pairs by selecting the best and worst candidates in a pool of responses to a given query. Empirically, we find that this approach improves the performance of any reward model, with an effect comparable to the addition of a similar quantity of human preference data. This work opens up new avenues of research for improving RLHF for language model alignment, by offering synthetic preference generation as a solution to reward modeling challenges. |
| title | West-of-N: Synthetic Preferences for Self-Improving Reward Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2401.12086 |