Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.08309 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917613599195136 |
|---|---|
| author | Li, Ang Xiao, Qiugen Cao, Peng Tang, Jian Yuan, Yi Zhao, Zijie Chen, Xiaoyuan Zhang, Liang Li, Xiangyang Yang, Kaitong Guo, Weidong Gan, Yukang Yu, Xu Wang, Daniell Shan, Ying |
| author_facet | Li, Ang Xiao, Qiugen Cao, Peng Tang, Jian Yuan, Yi Zhao, Zijie Chen, Xiaoyuan Zhang, Liang Li, Xiangyang Yang, Kaitong Guo, Weidong Gan, Yukang Yu, Xu Wang, Daniell Shan, Ying |
| contents | Reinforcement Learning from AI Feedback (RLAIF) has the advantages of shorter annotation cycles and lower costs over Reinforcement Learning from Human Feedback (RLHF), making it highly efficient during the rapid strategy iteration periods of large language model (LLM) training. Using ChatGPT as a labeler to provide feedback on open-domain prompts in RLAIF training, we observe an increase in human evaluators' preference win ratio for model responses, but a decrease in evaluators' satisfaction rate. Analysis suggests that the decrease in satisfaction rate is mainly due to some responses becoming less helpful, particularly in terms of correctness and truthfulness, highlighting practical limitations of basic RLAIF. In this paper, we propose Hybrid Reinforcement Learning from AI Feedback (HRLAIF). This method enhances the accuracy of AI annotations for responses, making the model's helpfulness more robust in training process. Additionally, it employs AI for Red Teaming, further improving the model's harmlessness. Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses. Compared to the policy model before Reinforcement Learning (RL), it achieves an increase of 2.08\% in satisfaction rate, effectively addressing the issue of a decrease of 4.58\% in satisfaction rate after basic RLAIF. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_08309 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback Li, Ang Xiao, Qiugen Cao, Peng Tang, Jian Yuan, Yi Zhao, Zijie Chen, Xiaoyuan Zhang, Liang Li, Xiangyang Yang, Kaitong Guo, Weidong Gan, Yukang Yu, Xu Wang, Daniell Shan, Ying Machine Learning Artificial Intelligence Reinforcement Learning from AI Feedback (RLAIF) has the advantages of shorter annotation cycles and lower costs over Reinforcement Learning from Human Feedback (RLHF), making it highly efficient during the rapid strategy iteration periods of large language model (LLM) training. Using ChatGPT as a labeler to provide feedback on open-domain prompts in RLAIF training, we observe an increase in human evaluators' preference win ratio for model responses, but a decrease in evaluators' satisfaction rate. Analysis suggests that the decrease in satisfaction rate is mainly due to some responses becoming less helpful, particularly in terms of correctness and truthfulness, highlighting practical limitations of basic RLAIF. In this paper, we propose Hybrid Reinforcement Learning from AI Feedback (HRLAIF). This method enhances the accuracy of AI annotations for responses, making the model's helpfulness more robust in training process. Additionally, it employs AI for Red Teaming, further improving the model's harmlessness. Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses. Compared to the policy model before Reinforcement Learning (RL), it achieves an increase of 2.08\% in satisfaction rate, effectively addressing the issue of a decrease of 4.58\% in satisfaction rate after basic RLAIF. |
| title | HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.08309 |