RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914932723810304 |
|---|---|
| author | Lee, Harrison Phatale, Samrat Mansoor, Hassan Mesnard, Thomas Ferret, Johan Lu, Kellie Bishop, Colton Hall, Ethan Carbune, Victor Rastogi, Abhinav Prakash, Sushant |
| author_facet | Lee, Harrison Phatale, Samrat Mansoor, Hassan Mesnard, Thomas Ferret, Johan Lu, Kellie Bishop, Colton Hall, Ethan Carbune, Victor Rastogi, Abhinav Prakash, Sushant |
| contents | Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards "self-improvement" by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_00267 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback Lee, Harrison Phatale, Samrat Mansoor, Hassan Mesnard, Thomas Ferret, Johan Lu, Kellie Bishop, Colton Hall, Ethan Carbune, Victor Rastogi, Abhinav Prakash, Sushant Computation and Language Artificial Intelligence Machine Learning Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards "self-improvement" by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF. |
| title | RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2309.00267 |