When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909870335197184 |
|---|---|
| author | Dong, Yijiang River Hu, Tiancheng Liu, Yinhong Üstün, Ahmet Collier, Nigel |
| author_facet | Dong, Yijiang River Hu, Tiancheng Liu, Yinhong Üstün, Ahmet Collier, Nigel |
| contents | While Reinforcement Learning from Human Feedback (RLHF) is widely used to align Large Language Models (LLMs) with human preferences, it typically assumes homogeneous preferences across users, overlooking diverse human values and minority viewpoints. Although personalized preference learning addresses this by tailoring separate preferences for individual users, the field lacks standardized methods to assess its effectiveness. We present a multi-faceted evaluation framework that measures not only performance but also fairness, unintended effects, and adaptability across varying levels of preference divergence. Through extensive experiments comparing eight personalization methods across three preference datasets, we demonstrate that performance differences between methods could reach 36% when users strongly disagree, and personalization can introduce up to 20% safety misalignment. These findings highlight the critical need for holistic evaluation approaches to advance the development of more effective and inclusive preference learning systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19158 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning Dong, Yijiang River Hu, Tiancheng Liu, Yinhong Üstün, Ahmet Collier, Nigel Computation and Language Artificial Intelligence While Reinforcement Learning from Human Feedback (RLHF) is widely used to align Large Language Models (LLMs) with human preferences, it typically assumes homogeneous preferences across users, overlooking diverse human values and minority viewpoints. Although personalized preference learning addresses this by tailoring separate preferences for individual users, the field lacks standardized methods to assess its effectiveness. We present a multi-faceted evaluation framework that measures not only performance but also fairness, unintended effects, and adaptability across varying levels of preference divergence. Through extensive experiments comparing eight personalization methods across three preference datasets, we demonstrate that performance differences between methods could reach 36% when users strongly disagree, and personalization can introduce up to 20% safety misalignment. These findings highlight the critical need for holistic evaluation approaches to advance the development of more effective and inclusive preference learning systems. |
| title | When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.19158 |