Learning from negative feedback, or positive feedback or both
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913724013477888 |
|---|---|
| author | Abdolmaleki, Abbas Piot, Bilal Shahriari, Bobak Springenberg, Jost Tobias Hertweck, Tim Joshi, Rishabh Oh, Junhyuk Bloesch, Michael Lampe, Thomas Heess, Nicolas Buchli, Jonas Riedmiller, Martin |
| author_facet | Abdolmaleki, Abbas Piot, Bilal Shahriari, Bobak Springenberg, Jost Tobias Hertweck, Tim Joshi, Rishabh Oh, Junhyuk Bloesch, Michael Lampe, Thomas Heess, Nicolas Buchli, Jonas Riedmiller, Martin |
| contents | Existing preference optimization methods often assume scenarios where paired preference feedback (preferred/positive vs. dis-preferred/negative examples) is available. This requirement limits their applicability in scenarios where only unpaired feedback--for example, either positive or negative--is available. To address this, we introduce a novel approach that decouples learning from positive and negative feedback. This decoupling enables control over the influence of each feedback type and, importantly, allows learning even when only one feedback type is present. A key contribution is demonstrating stable learning from negative feedback alone, a capability not well-addressed by current methods. Our approach builds upon the probabilistic framework introduced in (Dayan and Hinton, 1997), which uses expectation-maximization (EM) to directly optimize the probability of positive outcomes (as opposed to classic expected reward maximization). We address a key limitation in current EM-based methods: they solely maximize the likelihood of positive examples, while neglecting negative ones. We show how to extend EM algorithms to explicitly incorporate negative examples, leading to a theoretically grounded algorithm that offers an intuitive and versatile way to learn from both positive and negative feedback. We evaluate our approach for training language models based on human feedback as well as training policies for sequential decision-making problems, where learned value functions are available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04166 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning from negative feedback, or positive feedback or both Abdolmaleki, Abbas Piot, Bilal Shahriari, Bobak Springenberg, Jost Tobias Hertweck, Tim Joshi, Rishabh Oh, Junhyuk Bloesch, Michael Lampe, Thomas Heess, Nicolas Buchli, Jonas Riedmiller, Martin Machine Learning Existing preference optimization methods often assume scenarios where paired preference feedback (preferred/positive vs. dis-preferred/negative examples) is available. This requirement limits their applicability in scenarios where only unpaired feedback--for example, either positive or negative--is available. To address this, we introduce a novel approach that decouples learning from positive and negative feedback. This decoupling enables control over the influence of each feedback type and, importantly, allows learning even when only one feedback type is present. A key contribution is demonstrating stable learning from negative feedback alone, a capability not well-addressed by current methods. Our approach builds upon the probabilistic framework introduced in (Dayan and Hinton, 1997), which uses expectation-maximization (EM) to directly optimize the probability of positive outcomes (as opposed to classic expected reward maximization). We address a key limitation in current EM-based methods: they solely maximize the likelihood of positive examples, while neglecting negative ones. We show how to extend EM algorithms to explicitly incorporate negative examples, leading to a theoretically grounded algorithm that offers an intuitive and versatile way to learn from both positive and negative feedback. We evaluate our approach for training language models based on human feedback as well as training policies for sequential decision-making problems, where learned value functions are available. |
| title | Learning from negative feedback, or positive feedback or both |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.04166 |