Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929552728522752 |
|---|---|
| author | Zhang, Shun Chen, Zhenfang Chen, Sunli Shen, Yikang Sun, Zhiqing Gan, Chuang |
| author_facet | Zhang, Shun Chen, Zhenfang Chen, Sunli Shen, Yikang Sun, Zhiqing Gan, Chuang |
| contents | Reinforcement Learning from Human Feedback (RLHF) is a widely adopted approach for aligning large language models with human values. However, RLHF relies on a reward model that is trained with a limited amount of human preference data, which could lead to inaccurate predictions. As a result, RLHF may produce outputs that are misaligned with human values. To mitigate this issue, we contribute a reward ensemble method that allows the reward model to make more accurate predictions. As using an ensemble of large language model-based reward models can be computationally and resource-expensive, we explore efficient ensemble methods including linear-layer ensemble and LoRA-based ensemble. Empirically, we run Best-of-$n$ and Proximal Policy Optimization with our ensembled reward models, and verify that our ensemble methods help improve the alignment performance of RLHF outputs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_16635 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble Zhang, Shun Chen, Zhenfang Chen, Sunli Shen, Yikang Sun, Zhiqing Gan, Chuang Machine Learning Artificial Intelligence Computation and Language Reinforcement Learning from Human Feedback (RLHF) is a widely adopted approach for aligning large language models with human values. However, RLHF relies on a reward model that is trained with a limited amount of human preference data, which could lead to inaccurate predictions. As a result, RLHF may produce outputs that are misaligned with human values. To mitigate this issue, we contribute a reward ensemble method that allows the reward model to make more accurate predictions. As using an ensemble of large language model-based reward models can be computationally and resource-expensive, we explore efficient ensemble methods including linear-layer ensemble and LoRA-based ensemble. Empirically, we run Best-of-$n$ and Proximal Policy Optimization with our ensembled reward models, and verify that our ensemble methods help improve the alignment performance of RLHF outputs. |
| title | Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2401.16635 |