Prototypical Reward Network for Data-Efficient RLHF
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909245363978240 |
|---|---|
| author | Zhang, Jinghan Wang, Xiting Jin, Yiqiao Chen, Changyu Zhang, Xinhao Liu, Kunpeng |
| author_facet | Zhang, Jinghan Wang, Xiting Jin, Yiqiao Chen, Changyu Zhang, Xinhao Liu, Kunpeng |
| contents | The reward model for Reinforcement Learning from Human Feedback (RLHF) has proven effective in fine-tuning Large Language Models (LLMs). Notably, collecting human feedback for RLHF can be resource-intensive and lead to scalability issues for LLMs and complex tasks. Our proposed framework Proto-RM leverages prototypical networks to enhance reward models under limited human feedback. By enabling stable and reliable structural learning from fewer samples, Proto-RM significantly enhances LLMs' adaptability and accuracy in interpreting human preferences. Extensive experiments on various datasets demonstrate that Proto-RM significantly improves the performance of reward models and LLMs in human feedback tasks, achieving comparable and usually better results than traditional methods, while requiring significantly less data. in data-limited scenarios. This research offers a promising direction for enhancing the efficiency of reward models and optimizing the fine-tuning of language models under restricted feedback conditions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06606 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Prototypical Reward Network for Data-Efficient RLHF Zhang, Jinghan Wang, Xiting Jin, Yiqiao Chen, Changyu Zhang, Xinhao Liu, Kunpeng Computation and Language Artificial Intelligence The reward model for Reinforcement Learning from Human Feedback (RLHF) has proven effective in fine-tuning Large Language Models (LLMs). Notably, collecting human feedback for RLHF can be resource-intensive and lead to scalability issues for LLMs and complex tasks. Our proposed framework Proto-RM leverages prototypical networks to enhance reward models under limited human feedback. By enabling stable and reliable structural learning from fewer samples, Proto-RM significantly enhances LLMs' adaptability and accuracy in interpreting human preferences. Extensive experiments on various datasets demonstrate that Proto-RM significantly improves the performance of reward models and LLMs in human feedback tasks, achieving comparable and usually better results than traditional methods, while requiring significantly less data. in data-limited scenarios. This research offers a promising direction for enhancing the efficiency of reward models and optimizing the fine-tuning of language models under restricted feedback conditions. |
| title | Prototypical Reward Network for Data-Efficient RLHF |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.06606 |