Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.12116 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910020547903488 |
|---|---|
| author | Zhang, Pinyi Lin, Ting-En Wu, Yuchuan Chen, Jingyang Wang, Zongqi Yang, Hua Xu, Ze Huang, Fei Zhang, Kai Li, Yongbin |
| author_facet | Zhang, Pinyi Lin, Ting-En Wu, Yuchuan Chen, Jingyang Wang, Zongqi Yang, Hua Xu, Ze Huang, Fei Zhang, Kai Li, Yongbin |
| contents | Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_12116 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling Zhang, Pinyi Lin, Ting-En Wu, Yuchuan Chen, Jingyang Wang, Zongqi Yang, Hua Xu, Ze Huang, Fei Zhang, Kai Li, Yongbin Computation and Language Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability. |
| title | P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.12116 |