RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912308571144192 |
|---|---|
| author | Zhou, Enyu Zheng, Guodong Wang, Binghai Xi, Zhiheng Dou, Shihan Bao, Rong Shen, Wei Xiong, Limao Fan, Jessica Mou, Yurong Zheng, Rui Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Zhou, Enyu Zheng, Guodong Wang, Binghai Xi, Zhiheng Dou, Shihan Bao, Rong Shen, Wei Xiong, Limao Fan, Jessica Mou, Yurong Zheng, Rui Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | Reward models (RMs) guide the alignment of large language models (LLMs), steering them toward behaviors preferred by humans. Evaluating RMs is the key to better aligning LLMs. However, the current evaluation of RMs may not directly correspond to their alignment performance due to the limited distribution of evaluation data and evaluation methods that are not closely related to alignment objectives. To address these limitations, we propose RMB, a comprehensive RM benchmark that covers over 49 real-world scenarios and includes both pairwise and Best-of-N (BoN) evaluations to better reflect the effectiveness of RMs in guiding alignment optimization. We demonstrate a positive correlation between our benchmark and the downstream alignment task performance. Based on our benchmark, we conduct extensive analysis on the state-of-the-art RMs, revealing their generalization defects that were not discovered by previous benchmarks, and highlighting the potential of generative RMs. Furthermore, we delve into open questions in reward models, specifically examining the effectiveness of majority voting for the evaluation of reward models and analyzing the impact factors of generative RMs, including the influence of evaluation criteria and instructing methods. Our evaluation code and datasets are available at https://github.com/Zhou-Zoey/RMB-Reward-Model-Benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09893 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RMB: Comprehensively Benchmarking Reward Models in LLM Alignment Zhou, Enyu Zheng, Guodong Wang, Binghai Xi, Zhiheng Dou, Shihan Bao, Rong Shen, Wei Xiong, Limao Fan, Jessica Mou, Yurong Zheng, Rui Gui, Tao Zhang, Qi Huang, Xuanjing Computation and Language Reward models (RMs) guide the alignment of large language models (LLMs), steering them toward behaviors preferred by humans. Evaluating RMs is the key to better aligning LLMs. However, the current evaluation of RMs may not directly correspond to their alignment performance due to the limited distribution of evaluation data and evaluation methods that are not closely related to alignment objectives. To address these limitations, we propose RMB, a comprehensive RM benchmark that covers over 49 real-world scenarios and includes both pairwise and Best-of-N (BoN) evaluations to better reflect the effectiveness of RMs in guiding alignment optimization. We demonstrate a positive correlation between our benchmark and the downstream alignment task performance. Based on our benchmark, we conduct extensive analysis on the state-of-the-art RMs, revealing their generalization defects that were not discovered by previous benchmarks, and highlighting the potential of generative RMs. Furthermore, we delve into open questions in reward models, specifically examining the effectiveness of majority voting for the evaluation of reward models and analyzing the impact factors of generative RMs, including the influence of evaluation criteria and instructing methods. Our evaluation code and datasets are available at https://github.com/Zhou-Zoey/RMB-Reward-Model-Benchmark. |
| title | RMB: Comprehensively Benchmarking Reward Models in LLM Alignment |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.09893 |