HAF-RM: A Hybrid Alignment Framework for Reward Model Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912180115341312 |
|---|---|
| author | Liu, Shujun Shen, Xiaoyu Lai, Yuhang Wang, Siyuan Yue, Shengbin Huang, Zengfeng Huang, Xuanjing Wei, Zhongyu |
| author_facet | Liu, Shujun Shen, Xiaoyu Lai, Yuhang Wang, Siyuan Yue, Shengbin Huang, Zengfeng Huang, Xuanjing Wei, Zhongyu |
| contents | The reward model has become increasingly important in alignment, assessment, and data construction for large language models (LLMs). Most existing researchers focus on enhancing reward models through data improvements, following the conventional training framework for reward models that directly optimizes the predicted rewards. In this paper, we propose a hybrid alignment framework HaF-RM for reward model training by introducing an additional constraint on token-level policy probabilities in addition to the reward score. It can simultaneously supervise the internal preference model at the token level and optimize the mapping layer of the reward model at the sequence level. Experiment results on five datasets sufficiently show the validity and effectiveness of our proposed hybrid framework for training a high-quality reward model. By decoupling the reward modeling procedure and incorporating hybrid supervision, our HaF-RM framework offers a principled and effective approach to enhancing the performance and alignment of reward models, a critical component in the responsible development of powerful language models. We release our code at https://haf-rm.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_04185 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HAF-RM: A Hybrid Alignment Framework for Reward Model Training Liu, Shujun Shen, Xiaoyu Lai, Yuhang Wang, Siyuan Yue, Shengbin Huang, Zengfeng Huang, Xuanjing Wei, Zhongyu Computation and Language The reward model has become increasingly important in alignment, assessment, and data construction for large language models (LLMs). Most existing researchers focus on enhancing reward models through data improvements, following the conventional training framework for reward models that directly optimizes the predicted rewards. In this paper, we propose a hybrid alignment framework HaF-RM for reward model training by introducing an additional constraint on token-level policy probabilities in addition to the reward score. It can simultaneously supervise the internal preference model at the token level and optimize the mapping layer of the reward model at the sequence level. Experiment results on five datasets sufficiently show the validity and effectiveness of our proposed hybrid framework for training a high-quality reward model. By decoupling the reward modeling procedure and incorporating hybrid supervision, our HaF-RM framework offers a principled and effective approach to enhancing the performance and alignment of reward models, a critical component in the responsible development of powerful language models. We release our code at https://haf-rm.github.io. |
| title | HAF-RM: A Hybrid Alignment Framework for Reward Model Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.04185 |