ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914739106349056 |
|---|---|
| author | Hou, Zhenyu Niu, Yilin Du, Zhengxiao Zhang, Xiaohan Liu, Xiao Zeng, Aohan Zheng, Qinkai Huang, Minlie Wang, Hongning Tang, Jie Dong, Yuxiao |
| author_facet | Hou, Zhenyu Niu, Yilin Du, Zhengxiao Zhang, Xiaohan Liu, Xiao Zeng, Aohan Zheng, Qinkai Huang, Minlie Wang, Hongning Tang, Jie Dong, Yuxiao |
| contents | ChatGLM is a free-to-use AI service powered by the ChatGLM family of large language models (LLMs). In this paper, we present the ChatGLM-RLHF pipeline -- a reinforcement learning from human feedback (RLHF) system -- designed to enhance ChatGLM's alignment with human preferences. ChatGLM-RLHF encompasses three major components: the collection of human preference data, the training of the reward model, and the optimization of policies. Throughout the process of integrating ChatGLM-RLHF into production, we encountered and addressed several unprecedented challenges. We introduce the strategies to mitigate reward variance for stabilized large-scale training, implement model parallelism with fused gradient-descent, and design regularization constraints to avoid catastrophic forgetting in LLMs. Experiments show that ChatGLM-RLHF brings significant improvements in alignment tasks compared to the supervised fine-tuned (SFT) version of ChatGLM. For instance, it achieves on average 15\% more wins against ChatGLM-SFT in Chinese alignment tasks. The work presents our practices of aligning LLMs with human preferences, offering insights into the challenges and solutions in RLHF implementations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_00934 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback Hou, Zhenyu Niu, Yilin Du, Zhengxiao Zhang, Xiaohan Liu, Xiao Zeng, Aohan Zheng, Qinkai Huang, Minlie Wang, Hongning Tang, Jie Dong, Yuxiao Computation and Language ChatGLM is a free-to-use AI service powered by the ChatGLM family of large language models (LLMs). In this paper, we present the ChatGLM-RLHF pipeline -- a reinforcement learning from human feedback (RLHF) system -- designed to enhance ChatGLM's alignment with human preferences. ChatGLM-RLHF encompasses three major components: the collection of human preference data, the training of the reward model, and the optimization of policies. Throughout the process of integrating ChatGLM-RLHF into production, we encountered and addressed several unprecedented challenges. We introduce the strategies to mitigate reward variance for stabilized large-scale training, implement model parallelism with fused gradient-descent, and design regularization constraints to avoid catastrophic forgetting in LLMs. Experiments show that ChatGLM-RLHF brings significant improvements in alignment tasks compared to the supervised fine-tuned (SFT) version of ChatGLM. For instance, it achieves on average 15\% more wins against ChatGLM-SFT in Chinese alignment tasks. The work presents our practices of aligning LLMs with human preferences, offering insights into the challenges and solutions in RLHF implementations. |
| title | ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.00934 |