Generalized Preference Optimization: A Unified Approach to Offline Alignment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914815202557952 |
|---|---|
| author | Tang, Yunhao Guo, Zhaohan Daniel Zheng, Zeyu Calandriello, Daniele Munos, Rémi Rowland, Mark Richemond, Pierre Harvey Valko, Michal Pires, Bernardo Ávila Piot, Bilal |
| author_facet | Tang, Yunhao Guo, Zhaohan Daniel Zheng, Zeyu Calandriello, Daniele Munos, Rémi Rowland, Mark Richemond, Pierre Harvey Valko, Michal Pires, Bernardo Ávila Piot, Bilal |
| contents | Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss. Our analysis and experiments reveal the connections and subtle differences between the offline regularization and the KL divergence regularization intended by the canonical RLHF formulation. In a controlled setting akin to Gao et al 2023, we also show that different GPO variants achieve similar trade-offs between regularization and performance, though the optimal values of hyper-parameter might differ as predicted by theory. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05749 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generalized Preference Optimization: A Unified Approach to Offline Alignment Tang, Yunhao Guo, Zhaohan Daniel Zheng, Zeyu Calandriello, Daniele Munos, Rémi Rowland, Mark Richemond, Pierre Harvey Valko, Michal Pires, Bernardo Ávila Piot, Bilal Machine Learning Artificial Intelligence Offline preference optimization allows fine-tuning large models directly from offline data, and has proved effective in recent alignment practices. We propose generalized preference optimization (GPO), a family of offline losses parameterized by a general class of convex functions. GPO enables a unified view over preference optimization, encompassing existing algorithms such as DPO, IPO and SLiC as special cases, while naturally introducing new variants. The GPO framework also sheds light on how offline algorithms enforce regularization, through the design of the convex function that defines the loss. Our analysis and experiments reveal the connections and subtle differences between the offline regularization and the KL divergence regularization intended by the canonical RLHF formulation. In a controlled setting akin to Gao et al 2023, we also show that different GPO variants achieve similar trade-offs between regularization and performance, though the optimal values of hyper-parameter might differ as predicted by theory. In all, our results present new algorithmic toolkits and empirical insights to alignment practitioners. |
| title | Generalized Preference Optimization: A Unified Approach to Offline Alignment |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2402.05749 |