BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912049360011264 |
|---|---|
| author | Lee, Gihun Jeong, Minchan Kim, Yujin Jung, Hojung Oh, Jaehoon Kim, Sangmook Yun, Se-Young |
| author_facet | Lee, Gihun Jeong, Minchan Kim, Yujin Jung, Hojung Oh, Jaehoon Kim, Sangmook Yun, Se-Young |
| contents | While learning to align Large Language Models (LLMs) with human preferences has shown remarkable success, aligning these models to meet the diverse user preferences presents further challenges in preserving previous knowledge. This paper examines the impact of personalized preference optimization on LLMs, revealing that the extent of knowledge loss varies significantly with preference heterogeneity. Although previous approaches have utilized the KL constraint between the reference model and the policy model, we observe that they fail to maintain general knowledge and alignment when facing personalized preferences. To this end, we introduce Base-Anchored Preference Optimization (BAPO), a simple yet effective approach that utilizes the initial responses of reference model to mitigate forgetting while accommodating personalized alignment. BAPO effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment. Our experiments demonstrate the efficacy of BAPO in various setups. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00693 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization Lee, Gihun Jeong, Minchan Kim, Yujin Jung, Hojung Oh, Jaehoon Kim, Sangmook Yun, Se-Young Artificial Intelligence Computation and Language Machine Learning While learning to align Large Language Models (LLMs) with human preferences has shown remarkable success, aligning these models to meet the diverse user preferences presents further challenges in preserving previous knowledge. This paper examines the impact of personalized preference optimization on LLMs, revealing that the extent of knowledge loss varies significantly with preference heterogeneity. Although previous approaches have utilized the KL constraint between the reference model and the policy model, we observe that they fail to maintain general knowledge and alignment when facing personalized preferences. To this end, we introduce Base-Anchored Preference Optimization (BAPO), a simple yet effective approach that utilizes the initial responses of reference model to mitigate forgetting while accommodating personalized alignment. BAPO effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment. Our experiments demonstrate the efficacy of BAPO in various setups. |
| title | BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2407.00693 |