Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909483616174080 |
|---|---|
| author | Sun, Shengyang Zhang, Yian Bukharin, Alexander Mosallanezhad, David Zeng, Jiaqi Singhal, Soumye Shen, Gerald Renduchintala, Adithya Konuk, Tugrul Dong, Yi Wang, Zhilin Chichkov, Dmitry Delalleau, Olivier Kuchaiev, Oleksii |
| author_facet | Sun, Shengyang Zhang, Yian Bukharin, Alexander Mosallanezhad, David Zeng, Jiaqi Singhal, Soumye Shen, Gerald Renduchintala, Adithya Konuk, Tugrul Dong, Yi Wang, Zhilin Chichkov, Dmitry Delalleau, Olivier Kuchaiev, Oleksii |
| contents | The rapid development of large language model (LLM) alignment algorithms has resulted in a complex and fragmented landscape, with limited clarity on the effectiveness of different methods and their inter-connections. This paper introduces Reward-Aware Preference Optimization (RPO), a mathematical framework that unifies popular preference optimization techniques in LLM alignment, including DPO, IPO, SimPO, and REINFORCE (LOO), among others. RPO provides a structured approach to disentangle and systematically study the impact of various design choices, such as the optimization objective, the number of responses per prompt, and the use of implicit versus explicit reward models, on LLM preference optimization. We additionally propose a new experimental setup that enables the clean and direct ablation of such design choices. Through an extensive series of ablation studies within the RPO framework, we gain insights into the critical factors shaping model alignment, offering practical guidance on the most effective strategies for improving LLM alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00203 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment Sun, Shengyang Zhang, Yian Bukharin, Alexander Mosallanezhad, David Zeng, Jiaqi Singhal, Soumye Shen, Gerald Renduchintala, Adithya Konuk, Tugrul Dong, Yi Wang, Zhilin Chichkov, Dmitry Delalleau, Olivier Kuchaiev, Oleksii Machine Learning Computation and Language The rapid development of large language model (LLM) alignment algorithms has resulted in a complex and fragmented landscape, with limited clarity on the effectiveness of different methods and their inter-connections. This paper introduces Reward-Aware Preference Optimization (RPO), a mathematical framework that unifies popular preference optimization techniques in LLM alignment, including DPO, IPO, SimPO, and REINFORCE (LOO), among others. RPO provides a structured approach to disentangle and systematically study the impact of various design choices, such as the optimization objective, the number of responses per prompt, and the use of implicit versus explicit reward models, on LLM preference optimization. We additionally propose a new experimental setup that enables the clean and direct ablation of such design choices. Through an extensive series of ablation studies within the RPO framework, we gain insights into the critical factors shaping model alignment, offering practical guidance on the most effective strategies for improving LLM alignment. |
| title | Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.00203 |