Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.13210 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910510637645824 |
|---|---|
| author | Yang, Adam X. Robeyns, Maxime Coste, Thomas Shi, Zhengyan Wang, Jun Bou-Ammar, Haitham Aitchison, Laurence |
| author_facet | Yang, Adam X. Robeyns, Maxime Coste, Thomas Shi, Zhengyan Wang, Jun Bou-Ammar, Haitham Aitchison, Laurence |
| contents | To ensure that large language model (LLM) responses are helpful and non-toxic, a reward model trained on human preference data is usually used. LLM responses with high rewards are then selected through best-of-$n$ (BoN) sampling or the LLM is further optimized to produce responses with high rewards through reinforcement learning from human feedback (RLHF). However, these processes are susceptible to reward overoptimization or `hacking', where responses receive high rewards due to imperfections in the reward model rather than true preference, particularly as prompts or responses deviate from the training data. To address these challenges, we propose to train a Bayesian reward model, which signals higher uncertainty further from the training data distribution. We trained Bayesian reward models using Laplace approximation on LoRA weights, and found that the resulting uncertainty estimates can effectively mitigate reward overoptimization in BoN sampling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_13210 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bayesian Reward Models for LLM Alignment Yang, Adam X. Robeyns, Maxime Coste, Thomas Shi, Zhengyan Wang, Jun Bou-Ammar, Haitham Aitchison, Laurence Machine Learning To ensure that large language model (LLM) responses are helpful and non-toxic, a reward model trained on human preference data is usually used. LLM responses with high rewards are then selected through best-of-$n$ (BoN) sampling or the LLM is further optimized to produce responses with high rewards through reinforcement learning from human feedback (RLHF). However, these processes are susceptible to reward overoptimization or `hacking', where responses receive high rewards due to imperfections in the reward model rather than true preference, particularly as prompts or responses deviate from the training data. To address these challenges, we propose to train a Bayesian reward model, which signals higher uncertainty further from the training data distribution. We trained Bayesian reward models using Laplace approximation on LoRA weights, and found that the resulting uncertainty estimates can effectively mitigate reward overoptimization in BoN sampling. |
| title | Bayesian Reward Models for LLM Alignment |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2402.13210 |