Towards Comprehensive Preference Data Collection for Reward Modeling
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910499753426944 |
|---|---|
| author | Hu, Yulan Li, Qingyang Ouyang, Sheng Chen, Ge Chen, Kaihui Mei, Lijun Ye, Xucheng Zhang, Fuzheng Liu, Yong |
| author_facet | Hu, Yulan Li, Qingyang Ouyang, Sheng Chen, Ge Chen, Kaihui Mei, Lijun Ye, Xucheng Zhang, Fuzheng Liu, Yong |
| contents | Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models (LLMs) with human preferences, thereby enhancing the quality of responses generated. A critical component of RLHF is the reward model, which is trained on preference data and outputs a scalar reward during the inference stage. However, the collection of preference data still lacks thorough investigation. Recent studies indicate that preference data is collected either by AI or humans, where chosen and rejected instances are identified among pairwise responses. We question whether this process effectively filters out noise and ensures sufficient diversity in collected data. To address these concerns, for the first time, we propose a comprehensive framework for preference data collection, decomposing the process into four incremental steps: Prompt Generation, Response Generation, Response Filtering, and Human Labeling. This structured approach ensures the collection of high-quality preferences while reducing reliance on human labor. We conducted comprehensive experiments based on the data collected at different stages, demonstrating the effectiveness of the proposed data collection method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16486 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Comprehensive Preference Data Collection for Reward Modeling Hu, Yulan Li, Qingyang Ouyang, Sheng Chen, Ge Chen, Kaihui Mei, Lijun Ye, Xucheng Zhang, Fuzheng Liu, Yong Artificial Intelligence Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models (LLMs) with human preferences, thereby enhancing the quality of responses generated. A critical component of RLHF is the reward model, which is trained on preference data and outputs a scalar reward during the inference stage. However, the collection of preference data still lacks thorough investigation. Recent studies indicate that preference data is collected either by AI or humans, where chosen and rejected instances are identified among pairwise responses. We question whether this process effectively filters out noise and ensures sufficient diversity in collected data. To address these concerns, for the first time, we propose a comprehensive framework for preference data collection, decomposing the process into four incremental steps: Prompt Generation, Response Generation, Response Filtering, and Human Labeling. This structured approach ensures the collection of high-quality preferences while reducing reliance on human labor. We conducted comprehensive experiments based on the data collected at different stages, demonstrating the effectiveness of the proposed data collection method. |
| title | Towards Comprehensive Preference Data Collection for Reward Modeling |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2406.16486 |