MetaRM: Shifted Distributions Alignment via Meta-Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909186314469376 |
|---|---|
| author | Dou, Shihan Liu, Yan Zhou, Enyu Li, Tianlong Jia, Haoxiang Xiong, Limao Zhao, Xin Ye, Junjie Zheng, Rui Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Dou, Shihan Liu, Yan Zhou, Enyu Li, Tianlong Jia, Haoxiang Xiong, Limao Zhao, Xin Ye, Junjie Zheng, Rui Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | The success of Reinforcement Learning from Human Feedback (RLHF) in language model alignment is critically dependent on the capability of the reward model (RM). However, as the training process progresses, the output distribution of the policy model shifts, leading to the RM's reduced ability to distinguish between responses. This issue is further compounded when the RM, trained on a specific data distribution, struggles to generalize to examples outside of that distribution. These two issues can be united as a challenge posed by the shifted distribution of the environment. To surmount this challenge, we introduce MetaRM, a method leveraging meta-learning to align the RM with the shifted environment distribution. MetaRM is designed to train the RM by minimizing data loss, particularly for data that can improve the differentiation ability to examples of the shifted target distribution. Extensive experiments demonstrate that MetaRM significantly improves the RM's distinguishing ability in iterative RLHF optimization, and also provides the capacity to identify subtle differences in out-of-distribution samples. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_00438 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MetaRM: Shifted Distributions Alignment via Meta-Learning Dou, Shihan Liu, Yan Zhou, Enyu Li, Tianlong Jia, Haoxiang Xiong, Limao Zhao, Xin Ye, Junjie Zheng, Rui Gui, Tao Zhang, Qi Huang, Xuanjing Machine Learning Computation and Language The success of Reinforcement Learning from Human Feedback (RLHF) in language model alignment is critically dependent on the capability of the reward model (RM). However, as the training process progresses, the output distribution of the policy model shifts, leading to the RM's reduced ability to distinguish between responses. This issue is further compounded when the RM, trained on a specific data distribution, struggles to generalize to examples outside of that distribution. These two issues can be united as a challenge posed by the shifted distribution of the environment. To surmount this challenge, we introduce MetaRM, a method leveraging meta-learning to align the RM with the shifted environment distribution. MetaRM is designed to train the RM by minimizing data loss, particularly for data that can improve the differentiation ability to examples of the shifted target distribution. Extensive experiments demonstrate that MetaRM significantly improves the RM's distinguishing ability in iterative RLHF optimization, and also provides the capacity to identify subtle differences in out-of-distribution samples. |
| title | MetaRM: Shifted Distributions Alignment via Meta-Learning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2405.00438 |