Active Learning for Direct Preference Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912255905366016 |
|---|---|
| author | Kveton, Branislav Li, Xintong McAuley, Julian Rossi, Ryan Shang, Jingbo Wu, Junda Yu, Tong |
| author_facet | Kveton, Branislav Li, Xintong McAuley, Julian Rossi, Ryan Shang, Jingbo Wu, Junda Yu, Tong |
| contents | Direct preference optimization (DPO) is a form of reinforcement learning from human feedback (RLHF) where the policy is learned directly from preferential feedback. Although many models of human preferences exist, the critical task of selecting the most informative feedback for training them is under-explored. We propose an active learning framework for DPO, which can be applied to collect human feedback online or to choose the most informative subset of already collected feedback offline. We propose efficient algorithms for both settings. The key idea is to linearize the DPO objective at the last layer of the neural network representation of the optimized policy and then compute the D-optimal design to collect preferential feedback. We prove that the errors in our DPO logit estimates diminish with more feedback. We show the effectiveness of our algorithms empirically in the setting that matches our theory and also on large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01076 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Active Learning for Direct Preference Optimization Kveton, Branislav Li, Xintong McAuley, Julian Rossi, Ryan Shang, Jingbo Wu, Junda Yu, Tong Machine Learning Direct preference optimization (DPO) is a form of reinforcement learning from human feedback (RLHF) where the policy is learned directly from preferential feedback. Although many models of human preferences exist, the critical task of selecting the most informative feedback for training them is under-explored. We propose an active learning framework for DPO, which can be applied to collect human feedback online or to choose the most informative subset of already collected feedback offline. We propose efficient algorithms for both settings. The key idea is to linearize the DPO objective at the last layer of the neural network representation of the optimized policy and then compute the D-optimal design to collect preferential feedback. We prove that the errors in our DPO logit estimates diminish with more feedback. We show the effectiveness of our algorithms empirically in the setting that matches our theory and also on large language models. |
| title | Active Learning for Direct Preference Optimization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.01076 |