Sample Efficient Preference Alignment in LLMs via Active Exploration
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910884669947904 |
|---|---|
| author | Mehta, Viraj Belakaria, Syrine Das, Vikramjeet Neopane, Ojash Dai, Yijia Bogunovic, Ilija Engelhardt, Barbara Ermon, Stefano Schneider, Jeff Neiswanger, Willie |
| author_facet | Mehta, Viraj Belakaria, Syrine Das, Vikramjeet Neopane, Ojash Dai, Yijia Bogunovic, Ilija Engelhardt, Barbara Ermon, Stefano Schneider, Jeff Neiswanger, Willie |
| contents | Preference-based feedback is important for many applications in machine learning where evaluation of a reward function is not feasible. Notable recent examples arise in preference alignment for large language models, including in reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). For many applications of preference alignment, the cost of acquiring human feedback can be substantial. In this work, we take advantage of the fact that one can often choose contexts at which to obtain human feedback to most efficiently identify a good policy, and formalize the setting as an active contextual dueling bandit problem. We propose an active exploration algorithm to efficiently select the data and provide theoretical proof that it has a polynomial worst-case regret bound. We extend the setting and methodology for practical use in preference alignment of large language models. We provide two extensions, an online and an offline approach. Our method outperforms the baselines with limited samples of human preferences on several language models and four real-world datasets including two new datasets that we contribute to the literature. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_00267 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Sample Efficient Preference Alignment in LLMs via Active Exploration Mehta, Viraj Belakaria, Syrine Das, Vikramjeet Neopane, Ojash Dai, Yijia Bogunovic, Ilija Engelhardt, Barbara Ermon, Stefano Schneider, Jeff Neiswanger, Willie Machine Learning Artificial Intelligence Preference-based feedback is important for many applications in machine learning where evaluation of a reward function is not feasible. Notable recent examples arise in preference alignment for large language models, including in reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). For many applications of preference alignment, the cost of acquiring human feedback can be substantial. In this work, we take advantage of the fact that one can often choose contexts at which to obtain human feedback to most efficiently identify a good policy, and formalize the setting as an active contextual dueling bandit problem. We propose an active exploration algorithm to efficiently select the data and provide theoretical proof that it has a polynomial worst-case regret bound. We extend the setting and methodology for practical use in preference alignment of large language models. We provide two extensions, an online and an offline approach. Our method outperforms the baselines with limited samples of human preferences on several language models and four real-world datasets including two new datasets that we contribute to the literature. |
| title | Sample Efficient Preference Alignment in LLMs via Active Exploration |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2312.00267 |