Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2405.00747 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909335176609792 |
|---|---|
| author | Sharifnassab, Arsalan Salehkaleybar, Saber Ghiassian, Sina Kanoria, Surya Schuurmans, Dale |
| author_facet | Sharifnassab, Arsalan Salehkaleybar, Saber Ghiassian, Sina Kanoria, Surya Schuurmans, Dale |
| contents | We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_00747 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Soft Preference Optimization: Aligning Language Models to Expert Distributions Sharifnassab, Arsalan Salehkaleybar, Saber Ghiassian, Sina Kanoria, Surya Schuurmans, Dale Machine Learning Artificial Intelligence We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision. |
| title | Soft Preference Optimization: Aligning Language Models to Expert Distributions |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.00747 |