Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929554914803712 |
|---|---|
| author | Lin, Muhan Shi, Shuyang Guo, Yue Chalaki, Behdad Tadiparthi, Vaishnav Pari, Ehsan Moradi Stepputtis, Simon Campbell, Joseph Sycara, Katia |
| author_facet | Lin, Muhan Shi, Shuyang Guo, Yue Chalaki, Behdad Tadiparthi, Vaishnav Pari, Ehsan Moradi Stepputtis, Simon Campbell, Joseph Sycara, Katia |
| contents | The correct specification of reward models is a well-known challenge in reinforcement learning. Hand-crafted reward functions often lead to inefficient or suboptimal policies and may not be aligned with user values. Reinforcement learning from human feedback is a successful technique that can mitigate such issues, however, the collection of human feedback can be laborious. Recent works have solicited feedback from pre-trained large language models rather than humans to reduce or eliminate human effort, however, these approaches yield poor performance in the presence of hallucination and other errors. This paper studies the advantages and limitations of reinforcement learning from large language model feedback and proposes a simple yet effective method for soliciting and applying feedback as a potential-based shaping function. We theoretically show that inconsistent rankings, which approximate ranking errors, lead to uninformative rewards with our approach. Our method empirically improves convergence speed and policy returns over commonly used baselines even with significant ranking errors, and eliminates the need for complex post-processing of reward functions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_17389 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models Lin, Muhan Shi, Shuyang Guo, Yue Chalaki, Behdad Tadiparthi, Vaishnav Pari, Ehsan Moradi Stepputtis, Simon Campbell, Joseph Sycara, Katia Artificial Intelligence The correct specification of reward models is a well-known challenge in reinforcement learning. Hand-crafted reward functions often lead to inefficient or suboptimal policies and may not be aligned with user values. Reinforcement learning from human feedback is a successful technique that can mitigate such issues, however, the collection of human feedback can be laborious. Recent works have solicited feedback from pre-trained large language models rather than humans to reduce or eliminate human effort, however, these approaches yield poor performance in the presence of hallucination and other errors. This paper studies the advantages and limitations of reinforcement learning from large language model feedback and proposes a simple yet effective method for soliciting and applying feedback as a potential-based shaping function. We theoretically show that inconsistent rankings, which approximate ranking errors, lead to uninformative rewards with our approach. Our method empirically improves convergence speed and policy returns over commonly used baselines even with significant ranking errors, and eliminates the need for complex post-processing of reward functions. |
| title | Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2410.17389 |