Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917647600320512 |
|---|---|
| author | Xu, Jing Lee, Andrew Sukhbaatar, Sainbayar Weston, Jason |
| author_facet | Xu, Jing Lee, Andrew Sukhbaatar, Sainbayar Weston, Jason |
| contents | Practitioners commonly align large language models using pairwise preferences, i.e., given labels of the type response A is preferred to response B for a given input. Perhaps less commonly, methods have also been developed for binary feedback, i.e. training models given labels of type response A is good or bad. We show how an existing performant binary feedback method, the Cringe Loss (Adolphs et al., 2022), can be generalized to the pairwise preference setting using a simple soft margin extension. Pairwise Cringe Loss is straightforward to implement and efficient to train, and we find it outperforms state-of-the-art preference optimization algorithms such as PPO and DPO on the AlpacaFarm benchmark. We show that iterations of training of our model are important for improved results, and that we can generalize DPO to Iterative DPO in the same way. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_16682 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss Xu, Jing Lee, Andrew Sukhbaatar, Sainbayar Weston, Jason Computation and Language Artificial Intelligence Practitioners commonly align large language models using pairwise preferences, i.e., given labels of the type response A is preferred to response B for a given input. Perhaps less commonly, methods have also been developed for binary feedback, i.e. training models given labels of type response A is good or bad. We show how an existing performant binary feedback method, the Cringe Loss (Adolphs et al., 2022), can be generalized to the pairwise preference setting using a simple soft margin extension. Pairwise Cringe Loss is straightforward to implement and efficient to train, and we find it outperforms state-of-the-art preference optimization algorithms such as PPO and DPO on the AlpacaFarm benchmark. We show that iterations of training of our model are important for improved results, and that we can generalize DPO to Iterative DPO in the same way. |
| title | Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2312.16682 |