Exploring and Addressing Reward Confusion in Offline Preference Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909349183488000 |
|---|---|
| author | Chen, Xin Toyer, Sam Shkurti, Florian |
| author_facet | Chen, Xin Toyer, Sam Shkurti, Florian |
| contents | Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. This paper shows that offline RLHF is susceptible to reward confusion, especially in the presence of spurious correlations in offline data. We create a benchmark to study this problem and propose a method that can significantly reduce reward confusion by leveraging transitivity of preferences while building a global preference chain with active learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_16025 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring and Addressing Reward Confusion in Offline Preference Learning Chen, Xin Toyer, Sam Shkurti, Florian Machine Learning Artificial Intelligence Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. This paper shows that offline RLHF is susceptible to reward confusion, especially in the presence of spurious correlations in offline data. We create a benchmark to study this problem and propose a method that can significantly reduce reward confusion by leveraging transitivity of preferences while building a global preference chain with active learning. |
| title | Exploring and Addressing Reward Confusion in Offline Preference Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2407.16025 |