Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910826868244480 |
|---|---|
| author | Wen, Xueru Lou, Jie Lu, Yaojie Lin, Hongyu Yu, Xing Lu, Xinyu He, Ben Han, Xianpei Zhang, Debing Sun, Le |
| author_facet | Wen, Xueru Lou, Jie Lu, Yaojie Lin, Hongyu Yu, Xing Lu, Xinyu He, Ben Han, Xianpei Zhang, Debing Sun, Le |
| contents | Reward Models (RMs) are crucial for aligning language models with human preferences. Currently, the evaluation of RMs depends on measuring accuracy against a validation set of manually annotated preference data. Although this method is straightforward and widely adopted, the relationship between RM accuracy and downstream policy performance remains under-explored. In this work, we conduct experiments in a synthetic setting to investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance. Our findings reveal that while there is a weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance. Moreover, we discover that the way of measuring accuracy significantly impacts its ability to predict the final policy performance. Through the lens of the Regressional Goodhart effect, we recognize that accuracy, when used for measuring RM quality, can fail to fully capture the potential RM overoptimization. This underscores the inadequacy of relying solely on accuracy to reflect their impact on policy optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_05584 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? Wen, Xueru Lou, Jie Lu, Yaojie Lin, Hongyu Yu, Xing Lu, Xinyu He, Ben Han, Xianpei Zhang, Debing Sun, Le Machine Learning Artificial Intelligence Computation and Language Reward Models (RMs) are crucial for aligning language models with human preferences. Currently, the evaluation of RMs depends on measuring accuracy against a validation set of manually annotated preference data. Although this method is straightforward and widely adopted, the relationship between RM accuracy and downstream policy performance remains under-explored. In this work, we conduct experiments in a synthetic setting to investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance. Our findings reveal that while there is a weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance. Moreover, we discover that the way of measuring accuracy significantly impacts its ability to predict the final policy performance. Through the lens of the Regressional Goodhart effect, we recognize that accuracy, when used for measuring RM quality, can fail to fully capture the potential RM overoptimization. This underscores the inadequacy of relying solely on accuracy to reflect their impact on policy optimization. |
| title | Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2410.05584 |