Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.14076 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909744208281600 |
|---|---|
| author | Li, Mengdi Chen, Guanqiao Zhao, Xufeng Wen, Haochen Yang, Shu Wang, Di |
| author_facet | Li, Mengdi Chen, Guanqiao Zhao, Xufeng Wen, Haochen Yang, Shu Wang, Di |
| contents | Reward models (RMs), which are central to existing post-training methods, aim to align LLM outputs with human values by providing feedback signals during fine-tuning. However, existing RMs struggle to capture nuanced, user-specific preferences, especially under limited data and across diverse domains. Thus, we introduce PersRM-R1, the first reasoning-based reward modeling framework specifically designed to identify and represent personal factors from only one or a few personal exemplars. To address challenges including limited data availability and the requirement for robust generalization, our approach combines synthetic data generation with a two-stage training pipeline consisting of supervised fine-tuning followed by reinforcement fine-tuning. Experimental results demonstrate that PersRM-R1 outperforms existing models of similar size and matches the performance of much larger models in both accuracy and generalizability, paving the way for more effective personalized LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_14076 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning Li, Mengdi Chen, Guanqiao Zhao, Xufeng Wen, Haochen Yang, Shu Wang, Di Machine Learning Artificial Intelligence Reward models (RMs), which are central to existing post-training methods, aim to align LLM outputs with human values by providing feedback signals during fine-tuning. However, existing RMs struggle to capture nuanced, user-specific preferences, especially under limited data and across diverse domains. Thus, we introduce PersRM-R1, the first reasoning-based reward modeling framework specifically designed to identify and represent personal factors from only one or a few personal exemplars. To address challenges including limited data availability and the requirement for robust generalization, our approach combines synthetic data generation with a two-stage training pipeline consisting of supervised fine-tuning followed by reinforcement fine-tuning. Experimental results demonstrate that PersRM-R1 outperforms existing models of similar size and matches the performance of much larger models in both accuracy and generalizability, paving the way for more effective personalized LLMs. |
| title | PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.14076 |