Multi-turn Reinforcement Learning from Preference Human Feedback
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916502000631808 |
|---|---|
| author | Shani, Lior Rosenberg, Aviv Cassel, Asaf Lang, Oran Calandriello, Daniele Zipori, Avital Noga, Hila Keller, Orgad Piot, Bilal Szpektor, Idan Hassidim, Avinatan Matias, Yossi Munos, Rémi |
| author_facet | Shani, Lior Rosenberg, Aviv Cassel, Asaf Lang, Oran Calandriello, Daniele Zipori, Avital Noga, Hila Keller, Orgad Piot, Bilal Szpektor, Idan Hassidim, Avinatan Matias, Yossi Munos, Rémi |
| contents | Reinforcement Learning from Human Feedback (RLHF) has become the standard approach for aligning Large Language Models (LLMs) with human preferences, allowing LLMs to demonstrate remarkable abilities in various tasks. Existing methods work by emulating the preferences at the single decision (turn) level, limiting their capabilities in settings that require planning or multi-turn interactions to achieve a long-term goal. In this paper, we address this issue by developing novel methods for Reinforcement Learning (RL) from preference feedback between two full multi-turn conversations. In the tabular setting, we present a novel mirror-descent-based policy optimization algorithm for the general multi-turn preference-based RL problem, and prove its convergence to Nash equilibrium. To evaluate performance, we create a new environment, Education Dialogue, where a teacher agent guides a student in learning a random topic, and show that a deep RL variant of our algorithm outperforms RLHF baselines. Finally, we show that in an environment with explicit rewards, our algorithm recovers the same performance as a reward-based RL baseline, despite relying solely on a weaker preference signal. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14655 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-turn Reinforcement Learning from Preference Human Feedback Shani, Lior Rosenberg, Aviv Cassel, Asaf Lang, Oran Calandriello, Daniele Zipori, Avital Noga, Hila Keller, Orgad Piot, Bilal Szpektor, Idan Hassidim, Avinatan Matias, Yossi Munos, Rémi Machine Learning Reinforcement Learning from Human Feedback (RLHF) has become the standard approach for aligning Large Language Models (LLMs) with human preferences, allowing LLMs to demonstrate remarkable abilities in various tasks. Existing methods work by emulating the preferences at the single decision (turn) level, limiting their capabilities in settings that require planning or multi-turn interactions to achieve a long-term goal. In this paper, we address this issue by developing novel methods for Reinforcement Learning (RL) from preference feedback between two full multi-turn conversations. In the tabular setting, we present a novel mirror-descent-based policy optimization algorithm for the general multi-turn preference-based RL problem, and prove its convergence to Nash equilibrium. To evaluate performance, we create a new environment, Education Dialogue, where a teacher agent guides a student in learning a random topic, and show that a deep RL variant of our algorithm outperforms RLHF baselines. Finally, we show that in an environment with explicit rewards, our algorithm recovers the same performance as a reward-based RL baseline, despite relying solely on a weaker preference signal. |
| title | Multi-turn Reinforcement Learning from Preference Human Feedback |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.14655 |