Human Alignment of Large Language Models through Online Preference Optimisation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913264091267072 |
|---|---|
| author | Calandriello, Daniele Guo, Daniel Munos, Remi Rowland, Mark Tang, Yunhao Pires, Bernardo Avila Richemond, Pierre Harvey Lan, Charline Le Valko, Michal Liu, Tianqi Joshi, Rishabh Zheng, Zeyu Piot, Bilal |
| author_facet | Calandriello, Daniele Guo, Daniel Munos, Remi Rowland, Mark Tang, Yunhao Pires, Bernardo Avila Richemond, Pierre Harvey Lan, Charline Le Valko, Michal Liu, Tianqi Joshi, Rishabh Zheng, Zeyu Piot, Bilal |
| contents | Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD, that leverages the regularised sampling approach proposed by Nash-MD.
This equivalence may seem surprising at first sight, since IPO is an offline method whereas Nash-MD is an online method using a preference model. However, this equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. Optimising the IPO loss with such a stream of data becomes then equivalent to finding the Nash equilibrium of the preference model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm. We compare online-IPO and IPO-MD to different online versions of existing losses on preference data such as DPO and SLiC on a summarisation task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_08635 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Human Alignment of Large Language Models through Online Preference Optimisation Calandriello, Daniele Guo, Daniel Munos, Remi Rowland, Mark Tang, Yunhao Pires, Bernardo Avila Richemond, Pierre Harvey Lan, Charline Le Valko, Michal Liu, Tianqi Joshi, Rishabh Zheng, Zeyu Piot, Bilal Machine Learning Artificial Intelligence Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods such as Reinforcement Learning from Human Feedback (RLHF), Direct Policy Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In this paper, our contribution is two-fold. First, we show the equivalence between two recent alignment methods, namely Identity Policy Optimisation (IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation of IPO, named IPO-MD, that leverages the regularised sampling approach proposed by Nash-MD. This equivalence may seem surprising at first sight, since IPO is an offline method whereas Nash-MD is an online method using a preference model. However, this equivalence can be proven when we consider the online version of IPO, that is when both generations are sampled by the online policy and annotated by a trained preference model. Optimising the IPO loss with such a stream of data becomes then equivalent to finding the Nash equilibrium of the preference model through self-play. Building on this equivalence, we introduce the IPO-MD algorithm that generates data with a mixture policy (between the online and reference policy) similarly as the general Nash-MD algorithm. We compare online-IPO and IPO-MD to different online versions of existing losses on preference data such as DPO and SLiC on a summarisation task. |
| title | Human Alignment of Large Language Models through Online Preference Optimisation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.08635 |