Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.17534 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917801716875264 |
|---|---|
| author | Xu, Ruijie Liu, Zhihan Liu, Yongfei Yan, Shipeng Wang, Zhaoran Zhang, Zhi He, Xuming |
| author_facet | Xu, Ruijie Liu, Zhihan Liu, Yongfei Yan, Shipeng Wang, Zhaoran Zhang, Zhi He, Xuming |
| contents | We address the challenge of online Reinforcement Learning from Human Feedback (RLHF) with a focus on self-rewarding alignment methods. In online RLHF, obtaining feedback requires interaction with the environment, which can be costly when using additional reward models or the GPT-4 API. Current self-rewarding approaches rely heavily on the discriminator's judgment capabilities, which are effective for large-scale models but challenging to transfer to smaller ones. To address these limitations, we propose a novel, only-prompting self-rewarding online algorithm that generates preference datasets without relying on judgment capabilities. Additionally, we employ fine-grained arithmetic control over the optimality gap between positive and negative examples, generating more hard negatives in the later stages of training to help the model better capture subtle human preferences. Finally, we conduct extensive experiments on two base models, Mistral-7B and Mistral-Instruct-7B, which significantly bootstrap the performance of the reference model, achieving 34.5% in the Length-controlled Win Rates of AlpacaEval 2.0. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17534 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization Xu, Ruijie Liu, Zhihan Liu, Yongfei Yan, Shipeng Wang, Zhaoran Zhang, Zhi He, Xuming Artificial Intelligence We address the challenge of online Reinforcement Learning from Human Feedback (RLHF) with a focus on self-rewarding alignment methods. In online RLHF, obtaining feedback requires interaction with the environment, which can be costly when using additional reward models or the GPT-4 API. Current self-rewarding approaches rely heavily on the discriminator's judgment capabilities, which are effective for large-scale models but challenging to transfer to smaller ones. To address these limitations, we propose a novel, only-prompting self-rewarding online algorithm that generates preference datasets without relying on judgment capabilities. Additionally, we employ fine-grained arithmetic control over the optimality gap between positive and negative examples, generating more hard negatives in the later stages of training to help the model better capture subtle human preferences. Finally, we conduct extensive experiments on two base models, Mistral-7B and Mistral-Instruct-7B, which significantly bootstrap the performance of the reference model, achieving 34.5% in the Length-controlled Win Rates of AlpacaEval 2.0. |
| title | Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2409.17534 |