Reward Generalization in RLHF: A Topological Perspective
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909625611190272 |
|---|---|
| author | Qiu, Tianyi Zeng, Fanzhi Ji, Jiaming Yan, Dong Wang, Kaile Zhou, Jiayi Han, Yang Dai, Josef Pan, Xuehai Yang, Yaodong |
| author_facet | Qiu, Tianyi Zeng, Fanzhi Ji, Jiaming Yan, Dong Wang, Kaile Zhou, Jiayi Han, Yang Dai, Josef Pan, Xuehai Yang, Yaodong |
| contents | Existing alignment methods share a common topology of information flow, where reward information is collected from humans, modeled with preference learning, and used to tune language models. However, this shared topology has not been systematically characterized, nor have its alternatives been thoroughly explored, leaving the problems of low data efficiency and unreliable generalization unaddressed. As a solution, we introduce a theory of reward generalization in reinforcement learning from human feedback (RLHF), focusing on the topology of information flow at both macro and micro levels. At the macro level, we portray the RLHF information flow as an autoencoding process over behavior distributions, formalizing the RLHF objective of distributional consistency between human preference and model behavior. At the micro level, we present induced Bayesian networks to model the impact of dataset topologies on reward generalization. Combining analysis on both levels, we propose reward modeling from tree-structured preference information. It is shown to reduce reward uncertainty by up to $Θ(\log n/\log\log n)$ times compared to baselines, where $n$ is the dataset size. Validation on three NLP tasks shows that it achieves an average win rate of 65% against baselines, thus improving reward generalization for free via topology design, while reducing the amount of data requiring annotation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_10184 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reward Generalization in RLHF: A Topological Perspective Qiu, Tianyi Zeng, Fanzhi Ji, Jiaming Yan, Dong Wang, Kaile Zhou, Jiayi Han, Yang Dai, Josef Pan, Xuehai Yang, Yaodong Machine Learning Artificial Intelligence Computation and Language Discrete Mathematics Existing alignment methods share a common topology of information flow, where reward information is collected from humans, modeled with preference learning, and used to tune language models. However, this shared topology has not been systematically characterized, nor have its alternatives been thoroughly explored, leaving the problems of low data efficiency and unreliable generalization unaddressed. As a solution, we introduce a theory of reward generalization in reinforcement learning from human feedback (RLHF), focusing on the topology of information flow at both macro and micro levels. At the macro level, we portray the RLHF information flow as an autoencoding process over behavior distributions, formalizing the RLHF objective of distributional consistency between human preference and model behavior. At the micro level, we present induced Bayesian networks to model the impact of dataset topologies on reward generalization. Combining analysis on both levels, we propose reward modeling from tree-structured preference information. It is shown to reduce reward uncertainty by up to $Θ(\log n/\log\log n)$ times compared to baselines, where $n$ is the dataset size. Validation on three NLP tasks shows that it achieves an average win rate of 65% against baselines, thus improving reward generalization for free via topology design, while reducing the amount of data requiring annotation. |
| title | Reward Generalization in RLHF: A Topological Perspective |
| topic | Machine Learning Artificial Intelligence Computation and Language Discrete Mathematics |
| url | https://arxiv.org/abs/2402.10184 |