Averaging log-likelihoods in direct alignment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917707423678464 |
|---|---|
| author | Grinsztajn, Nathan Flet-Berliac, Yannis Azar, Mohammad Gheshlaghi Strub, Florian Wu, Bill Choi, Eugene Cremer, Chris Ahmadian, Arash Chandak, Yash Pietquin, Olivier Geist, Matthieu |
| author_facet | Grinsztajn, Nathan Flet-Berliac, Yannis Azar, Mohammad Gheshlaghi Strub, Florian Wu, Bill Choi, Eugene Cremer, Chris Ahmadian, Arash Chandak, Yash Pietquin, Olivier Geist, Matthieu |
| contents | To better align Large Language Models (LLMs) with human judgment, Reinforcement Learning from Human Feedback (RLHF) learns a reward model and then optimizes it using regularized RL. Recently, direct alignment methods were introduced to learn such a fine-tuned model directly from a preference dataset without computing a proxy reward function. These methods are built upon contrastive losses involving the log-likelihood of (dis)preferred completions according to the trained model. However, completions have various lengths, and the log-likelihood is not length-invariant. On the other side, the cross-entropy loss used in supervised training is length-invariant, as batches are typically averaged token-wise. To reconcile these approaches, we introduce a principled approach for making direct alignment length-invariant. Formally, we introduce a new averaging operator, to be composed with the optimality operator giving the best policy for the underlying RL problem. It translates into averaging the log-likelihood within the loss. We empirically study the effect of such averaging, observing a trade-off between the length of generations and their scores. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19188 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Averaging log-likelihoods in direct alignment Grinsztajn, Nathan Flet-Berliac, Yannis Azar, Mohammad Gheshlaghi Strub, Florian Wu, Bill Choi, Eugene Cremer, Chris Ahmadian, Arash Chandak, Yash Pietquin, Olivier Geist, Matthieu Machine Learning To better align Large Language Models (LLMs) with human judgment, Reinforcement Learning from Human Feedback (RLHF) learns a reward model and then optimizes it using regularized RL. Recently, direct alignment methods were introduced to learn such a fine-tuned model directly from a preference dataset without computing a proxy reward function. These methods are built upon contrastive losses involving the log-likelihood of (dis)preferred completions according to the trained model. However, completions have various lengths, and the log-likelihood is not length-invariant. On the other side, the cross-entropy loss used in supervised training is length-invariant, as batches are typically averaged token-wise. To reconcile these approaches, we introduce a principled approach for making direct alignment length-invariant. Formally, we introduce a new averaging operator, to be composed with the optimality operator giving the best policy for the underlying RL problem. It translates into averaging the log-likelihood within the loss. We empirically study the effect of such averaging, observing a trade-off between the length of generations and their scores. |
| title | Averaging log-likelihoods in direct alignment |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.19188 |