Neural Dueling Bandits: Preference-Based Optimization with Human Feedback
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913795908042752 |
|---|---|
| author | Verma, Arun Dai, Zhongxiang Lin, Xiaoqiang Jaillet, Patrick Low, Bryan Kian Hsiang |
| author_facet | Verma, Arun Dai, Zhongxiang Lin, Xiaoqiang Jaillet, Patrick Low, Bryan Kian Hsiang |
| contents | Contextual dueling bandit is used to model the bandit problems, where a learner's goal is to find the best arm for a given context using observed noisy human preference feedback over the selected arms for the past contexts. However, existing algorithms assume the reward function is linear, which can be complex and non-linear in many real-life applications like online recommendations or ranking web search results. To overcome this challenge, we use a neural network to estimate the reward function using preference feedback for the previously selected arms. We propose upper confidence bound- and Thompson sampling-based algorithms with sub-linear regret guarantees that efficiently select arms in each round. We also extend our theoretical results to contextual bandit problems with binary feedback, which is in itself a non-trivial contribution. Experimental results on the problem instances derived from synthetic datasets corroborate our theoretical results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_17112 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Neural Dueling Bandits: Preference-Based Optimization with Human Feedback Verma, Arun Dai, Zhongxiang Lin, Xiaoqiang Jaillet, Patrick Low, Bryan Kian Hsiang Machine Learning Artificial Intelligence Contextual dueling bandit is used to model the bandit problems, where a learner's goal is to find the best arm for a given context using observed noisy human preference feedback over the selected arms for the past contexts. However, existing algorithms assume the reward function is linear, which can be complex and non-linear in many real-life applications like online recommendations or ranking web search results. To overcome this challenge, we use a neural network to estimate the reward function using preference feedback for the previously selected arms. We propose upper confidence bound- and Thompson sampling-based algorithms with sub-linear regret guarantees that efficiently select arms in each round. We also extend our theoretical results to contextual bandit problems with binary feedback, which is in itself a non-trivial contribution. Experimental results on the problem instances derived from synthetic datasets corroborate our theoretical results. |
| title | Neural Dueling Bandits: Preference-Based Optimization with Human Feedback |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2407.17112 |