Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909905522262016 |
|---|---|
| author | Wang, Chenglong Huo, Yifu Gan, Yang Mu, Yongyu He, Qiaozhi Yang, Murun Li, Bei Zhang, Chunliang Liu, Tongran Ma, Anxiang Yu, Zhengtao Zhu, Jingbo Xiao, Tong |
| author_facet | Wang, Chenglong Huo, Yifu Gan, Yang Mu, Yongyu He, Qiaozhi Yang, Murun Li, Bei Zhang, Chunliang Liu, Tongran Ma, Anxiang Yu, Zhengtao Zhu, Jingbo Xiao, Tong |
| contents | Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with the alignment performance of large language models (LLMs), making it a reliable reference for developing advanced reward models. Our analysis of MRMBench evaluation results reveals that reward models often struggle to capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Additionally, our findings show that the proposed inference-time probing method offers a reliable metric for assessing the confidence of reward predictions, which ultimately improves the alignment of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12464 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models Wang, Chenglong Huo, Yifu Gan, Yang Mu, Yongyu He, Qiaozhi Yang, Murun Li, Bei Zhang, Chunliang Liu, Tongran Ma, Anxiang Yu, Zhengtao Zhu, Jingbo Xiao, Tong Computation and Language Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with the alignment performance of large language models (LLMs), making it a reliable reference for developing advanced reward models. Our analysis of MRMBench evaluation results reveals that reward models often struggle to capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Additionally, our findings show that the proposed inference-time probing method offers a reliable metric for assessing the confidence of reward predictions, which ultimately improves the alignment of LLMs. |
| title | Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.12464 |