Active Human Feedback Collection via Neural Contextual Dueling Bandits
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912487183482880 |
|---|---|
| author | Verma, Arun Lin, Xiaoqiang Dai, Zhongxiang Rus, Daniela Low, Bryan Kian Hsiang |
| author_facet | Verma, Arun Lin, Xiaoqiang Dai, Zhongxiang Rus, Daniela Low, Bryan Kian Hsiang |
| contents | Collecting human preference feedback is often expensive, leading recent works to develop principled algorithms to select them more efficiently. However, these works assume that the underlying reward function is linear, an assumption that does not hold in many real-life applications, such as online recommendation and LLM alignment. To address this limitation, we propose Neural-ADB, an algorithm based on the neural contextual dueling bandit framework that provides a principled and practical method for collecting human preference feedback when the underlying latent reward function is non-linear. We theoretically show that when preference feedback follows the Bradley-Terry-Luce model, the worst sub-optimality gap of the policy learned by Neural-ADB decreases at a sub-linear rate as the preference dataset increases. Our experimental results on preference datasets further corroborate the effectiveness of Neural-ADB. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_12016 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Active Human Feedback Collection via Neural Contextual Dueling Bandits Verma, Arun Lin, Xiaoqiang Dai, Zhongxiang Rus, Daniela Low, Bryan Kian Hsiang Machine Learning Collecting human preference feedback is often expensive, leading recent works to develop principled algorithms to select them more efficiently. However, these works assume that the underlying reward function is linear, an assumption that does not hold in many real-life applications, such as online recommendation and LLM alignment. To address this limitation, we propose Neural-ADB, an algorithm based on the neural contextual dueling bandit framework that provides a principled and practical method for collecting human preference feedback when the underlying latent reward function is non-linear. We theoretically show that when preference feedback follows the Bradley-Terry-Luce model, the worst sub-optimality gap of the policy learned by Neural-ADB decreases at a sub-linear rate as the preference dataset increases. Our experimental results on preference datasets further corroborate the effectiveness of Neural-ADB. |
| title | Active Human Feedback Collection via Neural Contextual Dueling Bandits |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.12016 |