Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfe
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910764827148288 |
|---|---|
| author | Thekumparampil, Kiran Koshy Hiranandani, Gaurush Kalantari, Kousha Sabach, Shoham Kveton, Branislav |
| author_facet | Thekumparampil, Kiran Koshy Hiranandani, Gaurush Kalantari, Kousha Sabach, Shoham Kveton, Branislav |
| contents | We study learning of human preferences from a limited comparison feedback. This task is ubiquitous in machine learning. Its applications such as reinforcement learning from human feedback, have been transformational. We formulate this problem as learning a Plackett-Luce model over a universe of $N$ choices from $K$-way comparison feedback, where typically $K \ll N$. Our solution is the D-optimal design for the Plackett-Luce objective. The design defines a data logging policy that elicits comparison feedback for a small collection of optimally chosen points from all ${N \choose K}$ feasible subsets. The main algorithmic challenge in this work is that even fast methods for solving D-optimal designs would have $O({N \choose K})$ time complexity. To address this issue, we propose a randomized Frank-Wolfe (FW) algorithm that solves the linear maximization sub-problems in the FW method on randomly chosen variables. We analyze the algorithm, and evaluate it empirically on synthetic and open-source NLP datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_19396 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfe Thekumparampil, Kiran Koshy Hiranandani, Gaurush Kalantari, Kousha Sabach, Shoham Kveton, Branislav Machine Learning Artificial Intelligence Information Theory Optimization and Control We study learning of human preferences from a limited comparison feedback. This task is ubiquitous in machine learning. Its applications such as reinforcement learning from human feedback, have been transformational. We formulate this problem as learning a Plackett-Luce model over a universe of $N$ choices from $K$-way comparison feedback, where typically $K \ll N$. Our solution is the D-optimal design for the Plackett-Luce objective. The design defines a data logging policy that elicits comparison feedback for a small collection of optimally chosen points from all ${N \choose K}$ feasible subsets. The main algorithmic challenge in this work is that even fast methods for solving D-optimal designs would have $O({N \choose K})$ time complexity. To address this issue, we propose a randomized Frank-Wolfe (FW) algorithm that solves the linear maximization sub-problems in the FW method on randomly chosen variables. We analyze the algorithm, and evaluate it empirically on synthetic and open-source NLP datasets. |
| title | Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfe |
| topic | Machine Learning Artificial Intelligence Information Theory Optimization and Control |
| url | https://arxiv.org/abs/2412.19396 |