CoPL: Collaborative Preference Learning for Personalizing LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912590361264128 |
|---|---|
| author | Choi, Youngbin Cho, Seunghyuk Lee, Minjong Park, MoonJeong Ko, Yesong Ok, Jungseul Kim, Dongwoo |
| author_facet | Choi, Youngbin Cho, Seunghyuk Lee, Minjong Park, MoonJeong Ko, Yesong Ok, Jungseul Kim, Dongwoo |
| contents | Personalizing large language models (LLMs) is important for aligning outputs with diverse user preferences, yet existing methods struggle with flexibility and generalization. We propose CoPL (Collaborative Preference Learning), a graph-based collaborative filtering framework that models user-response relationships to enhance preference estimation, particularly in sparse annotation settings. By integrating a mixture of LoRA experts, CoPL efficiently fine-tunes LLMs while dynamically balancing shared and user-specific preferences. Additionally, an optimization-free adaptation strategy enables generalization to unseen users without fine-tuning. Experiments on UltraFeedback-P demonstrate that CoPL outperforms existing personalized reward models, effectively capturing both common and controversial preferences, making it a scalable solution for personalized LLM alignment. The code is available at https://github.com/ml-postech/CoPL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01658 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CoPL: Collaborative Preference Learning for Personalizing LLMs Choi, Youngbin Cho, Seunghyuk Lee, Minjong Park, MoonJeong Ko, Yesong Ok, Jungseul Kim, Dongwoo Machine Learning Artificial Intelligence Information Retrieval Personalizing large language models (LLMs) is important for aligning outputs with diverse user preferences, yet existing methods struggle with flexibility and generalization. We propose CoPL (Collaborative Preference Learning), a graph-based collaborative filtering framework that models user-response relationships to enhance preference estimation, particularly in sparse annotation settings. By integrating a mixture of LoRA experts, CoPL efficiently fine-tunes LLMs while dynamically balancing shared and user-specific preferences. Additionally, an optimization-free adaptation strategy enables generalization to unseen users without fine-tuning. Experiments on UltraFeedback-P demonstrate that CoPL outperforms existing personalized reward models, effectively capturing both common and controversial preferences, making it a scalable solution for personalized LLM alignment. The code is available at https://github.com/ml-postech/CoPL. |
| title | CoPL: Collaborative Preference Learning for Personalizing LLMs |
| topic | Machine Learning Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2503.01658 |