Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929736942354432 |
|---|---|
| author | Zhang, Yuheng Yu, Dian Peng, Baolin Song, Linfeng Tian, Ye Huo, Mingyue Jiang, Nan Mi, Haitao Yu, Dong |
| author_facet | Zhang, Yuheng Yu, Dian Peng, Baolin Song, Linfeng Tian, Ye Huo, Mingyue Jiang, Nan Mi, Haitao Yu, Dong |
| contents | Reinforcement Learning with Human Feedback (RLHF) has achieved great success in aligning large language models (LLMs) with human preferences. Prevalent RLHF approaches are reward-based, following the Bradley-Terry (BT) model assumption, which may not fully capture the complexity of human preferences. In this paper, we explore RLHF under a general preference framework and approach it from a game-theoretic perspective. Specifically, we formulate the problem as a two-player game and propose a novel online algorithm, iterative Nash policy optimization (INPO). The key idea is to let the policy play against itself via no-regret learning, thereby approximating the Nash policy. Unlike previous methods, INPO bypasses the need for estimating the expected win rate for individual responses, which typically incurs high computational or annotation costs. Instead, we introduce a new loss objective that is directly minimized over a preference dataset. We provide theoretical analysis for our approach and demonstrate its effectiveness through experiments on various representative benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 42.6% length-controlled win rate on AlpacaEval 2.0 and a 37.8% win rate on Arena-Hard, showing substantial improvement over the state-of-the-art online RLHF algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00617 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning Zhang, Yuheng Yu, Dian Peng, Baolin Song, Linfeng Tian, Ye Huo, Mingyue Jiang, Nan Mi, Haitao Yu, Dong Machine Learning Artificial Intelligence Computation and Language Computer Science and Game Theory Reinforcement Learning with Human Feedback (RLHF) has achieved great success in aligning large language models (LLMs) with human preferences. Prevalent RLHF approaches are reward-based, following the Bradley-Terry (BT) model assumption, which may not fully capture the complexity of human preferences. In this paper, we explore RLHF under a general preference framework and approach it from a game-theoretic perspective. Specifically, we formulate the problem as a two-player game and propose a novel online algorithm, iterative Nash policy optimization (INPO). The key idea is to let the policy play against itself via no-regret learning, thereby approximating the Nash policy. Unlike previous methods, INPO bypasses the need for estimating the expected win rate for individual responses, which typically incurs high computational or annotation costs. Instead, we introduce a new loss objective that is directly minimized over a preference dataset. We provide theoretical analysis for our approach and demonstrate its effectiveness through experiments on various representative benchmarks. With an LLaMA-3-8B-based SFT model, INPO achieves a 42.6% length-controlled win rate on AlpacaEval 2.0 and a 37.8% win rate on Arena-Hard, showing substantial improvement over the state-of-the-art online RLHF algorithms. |
| title | Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Science and Game Theory |
| url | https://arxiv.org/abs/2407.00617 |