PrLM: Learning Explicit Reasoning for Personalized RAG via Contrastive Reward Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916890423590912 |
|---|---|
| author | Zhang, Kepu Shi, Teng Yu, Weijie Xu, Jun |
| author_facet | Zhang, Kepu Shi, Teng Yu, Weijie Xu, Jun |
| contents | Personalized retrieval-augmented generation (RAG) aims to produce user-tailored responses by incorporating retrieved user profiles alongside the input query. Existing methods primarily focus on improving retrieval and rely on large language models (LLMs) to implicitly integrate the retrieved context with the query. However, such models are often sensitive to retrieval quality and may generate responses that are misaligned with user preferences. To address this limitation, we propose PrLM, a reinforcement learning framework that trains LLMs to explicitly reason over retrieved user profiles. Guided by a contrastively trained personalization reward model, PrLM effectively learns from user responses without requiring annotated reasoning paths. Experiments on three personalized text generation datasets show that PrLM outperforms existing methods and remains robust across varying numbers of retrieved profiles and different retrievers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_07342 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PrLM: Learning Explicit Reasoning for Personalized RAG via Contrastive Reward Optimization Zhang, Kepu Shi, Teng Yu, Weijie Xu, Jun Information Retrieval Computation and Language Personalized retrieval-augmented generation (RAG) aims to produce user-tailored responses by incorporating retrieved user profiles alongside the input query. Existing methods primarily focus on improving retrieval and rely on large language models (LLMs) to implicitly integrate the retrieved context with the query. However, such models are often sensitive to retrieval quality and may generate responses that are misaligned with user preferences. To address this limitation, we propose PrLM, a reinforcement learning framework that trains LLMs to explicitly reason over retrieved user profiles. Guided by a contrastively trained personalization reward model, PrLM effectively learns from user responses without requiring annotated reasoning paths. Experiments on three personalized text generation datasets show that PrLM outperforms existing methods and remains robust across varying numbers of retrieved profiles and different retrievers. |
| title | PrLM: Learning Explicit Reasoning for Personalized RAG via Contrastive Reward Optimization |
| topic | Information Retrieval Computation and Language |
| url | https://arxiv.org/abs/2508.07342 |