Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.14439 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915251475185664 |
|---|---|
| author | Bose, Avinandan Xiong, Zhihan Chi, Yuejie Du, Simon Shaolei Xiao, Lin Fazel, Maryam |
| author_facet | Bose, Avinandan Xiong, Zhihan Chi, Yuejie Du, Simon Shaolei Xiao, Lin Fazel, Maryam |
| contents | Personalizing large language models (LLMs) to accommodate diverse user preferences is essential for enhancing alignment and user satisfaction. Traditional reinforcement learning from human feedback (RLHF) approaches often rely on monolithic value representations, limiting their ability to adapt to individual preferences. We introduce a novel framework that leverages low-rank preference modeling to efficiently learn and generalize user-specific reward functions. By representing reward functions in a low-dimensional subspace and modeling individual preferences as weighted combinations of shared basis functions, our approach avoids rigid user categorization while enabling scalability and few-shot adaptation. We validate our method on multiple preference datasets, demonstrating superior generalization to unseen users and improved accuracy in preference prediction tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14439 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LoRe: Personalizing LLMs via Low-Rank Reward Modeling Bose, Avinandan Xiong, Zhihan Chi, Yuejie Du, Simon Shaolei Xiao, Lin Fazel, Maryam Machine Learning Artificial Intelligence Computation and Language Personalizing large language models (LLMs) to accommodate diverse user preferences is essential for enhancing alignment and user satisfaction. Traditional reinforcement learning from human feedback (RLHF) approaches often rely on monolithic value representations, limiting their ability to adapt to individual preferences. We introduce a novel framework that leverages low-rank preference modeling to efficiently learn and generalize user-specific reward functions. By representing reward functions in a low-dimensional subspace and modeling individual preferences as weighted combinations of shared basis functions, our approach avoids rigid user categorization while enabling scalability and few-shot adaptation. We validate our method on multiple preference datasets, demonstrating superior generalization to unseen users and improved accuracy in preference prediction tasks. |
| title | LoRe: Personalizing LLMs via Low-Rank Reward Modeling |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2504.14439 |