Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908327494025216 |
|---|---|
| author | Wang, Mingzhi Ma, Chengdong Chen, Qizhi Meng, Linjian Han, Yang Xiao, Jiancong Zhang, Zhaowei Huo, Jing Su, Weijie J. Yang, Yaodong |
| author_facet | Wang, Mingzhi Ma, Chengdong Chen, Qizhi Meng, Linjian Han, Yang Xiao, Jiancong Zhang, Zhaowei Huo, Jing Su, Weijie J. Yang, Yaodong |
| contents | Self-play methods have demonstrated remarkable success in enhancing model capabilities across various domains. In the context of Reinforcement Learning from Human Feedback (RLHF), self-play not only boosts Large Language Model (LLM) performance but also overcomes the limitations of traditional Bradley-Terry (BT) model assumptions by finding the Nash equilibrium (NE) of a preference-based, two-player constant-sum game. However, existing methods either guarantee only average-iterate convergence, incurring high storage and inference costs, or converge to the NE of a regularized game, failing to accurately reflect true human preferences. In this paper, we introduce Magnetic Preference Optimization (MPO), a novel approach capable of achieving last-iterate convergence to the NE of the original game, effectively overcoming the limitations of existing methods. Building upon Magnetic Mirror Descent (MMD), MPO attains a linear convergence rate, making it particularly suitable for fine-tuning LLMs. To ensure our algorithm is both theoretically sound and practically viable, we present a simple yet effective implementation that adapts the theoretical insights to the RLHF setting. Empirical results demonstrate that MPO can significantly enhance the performance of LLMs, highlighting the potential of self-play methods in alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16714 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment Wang, Mingzhi Ma, Chengdong Chen, Qizhi Meng, Linjian Han, Yang Xiao, Jiancong Zhang, Zhaowei Huo, Jing Su, Weijie J. Yang, Yaodong Computation and Language Self-play methods have demonstrated remarkable success in enhancing model capabilities across various domains. In the context of Reinforcement Learning from Human Feedback (RLHF), self-play not only boosts Large Language Model (LLM) performance but also overcomes the limitations of traditional Bradley-Terry (BT) model assumptions by finding the Nash equilibrium (NE) of a preference-based, two-player constant-sum game. However, existing methods either guarantee only average-iterate convergence, incurring high storage and inference costs, or converge to the NE of a regularized game, failing to accurately reflect true human preferences. In this paper, we introduce Magnetic Preference Optimization (MPO), a novel approach capable of achieving last-iterate convergence to the NE of the original game, effectively overcoming the limitations of existing methods. Building upon Magnetic Mirror Descent (MMD), MPO attains a linear convergence rate, making it particularly suitable for fine-tuning LLMs. To ensure our algorithm is both theoretically sound and practically viable, we present a simple yet effective implementation that adapts the theoretical insights to the RLHF setting. Empirical results demonstrate that MPO can significantly enhance the performance of LLMs, highlighting the potential of self-play methods in alignment. |
| title | Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.16714 |