A Systematic Analysis of Base Model Choice for Reward Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910947374792704 |
|---|---|
| author | Ahrabian, Kian Jandaghi, Pegah Mokhberian, Negar Karimireddy, Sai Praneeth Pujara, Jay |
| author_facet | Ahrabian, Kian Jandaghi, Pegah Mokhberian, Negar Karimireddy, Sai Praneeth Pujara, Jay |
| contents | Reinforcement learning from human feedback (RLHF) and, at its core, reward modeling have become a crucial part of training powerful large language models (LLMs). One commonly overlooked factor in training high-quality reward models (RMs) is the effect of the base model, which is becoming more challenging to choose given the rapidly growing pool of LLMs. In this work, we present a systematic analysis of the effect of base model selection on reward modeling performance. Our results show that the performance can be improved by up to 14% compared to the most common (i.e., default) choice. Moreover, we showcase the strong statistical relation between some existing benchmarks and downstream performances. We also demonstrate that the results from a small set of benchmarks could be combined to boost the model selection ($+$18% on average in the top 5-10). Lastly, we illustrate the impact of different post-training steps on the final performance and explore using estimated data distributions to reduce performance prediction error. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_10775 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Systematic Analysis of Base Model Choice for Reward Modeling Ahrabian, Kian Jandaghi, Pegah Mokhberian, Negar Karimireddy, Sai Praneeth Pujara, Jay Computation and Language Artificial Intelligence Reinforcement learning from human feedback (RLHF) and, at its core, reward modeling have become a crucial part of training powerful large language models (LLMs). One commonly overlooked factor in training high-quality reward models (RMs) is the effect of the base model, which is becoming more challenging to choose given the rapidly growing pool of LLMs. In this work, we present a systematic analysis of the effect of base model selection on reward modeling performance. Our results show that the performance can be improved by up to 14% compared to the most common (i.e., default) choice. Moreover, we showcase the strong statistical relation between some existing benchmarks and downstream performances. We also demonstrate that the results from a small set of benchmarks could be combined to boost the model selection ($+$18% on average in the top 5-10). Lastly, we illustrate the impact of different post-training steps on the final performance and explore using estimated data distributions to reduce performance prediction error. |
| title | A Systematic Analysis of Base Model Choice for Reward Modeling |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.10775 |