Personalized Adaptation via In-Context Preference Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910656169508864 |
|---|---|
| author | Lau, Allison Choi, Younwoo Balazadeh, Vahid Chidambaram, Keertana Syrgkanis, Vasilis Krishnan, Rahul G. |
| author_facet | Lau, Allison Choi, Younwoo Balazadeh, Vahid Chidambaram, Keertana Syrgkanis, Vasilis Krishnan, Rahul G. |
| contents | Reinforcement Learning from Human Feedback (RLHF) is widely used to align Language Models (LMs) with human preferences. However, existing approaches often neglect individual user preferences, leading to suboptimal personalization. We present the Preference Pretrained Transformer (PPT), a novel approach for adaptive personalization using online user feedback. PPT leverages the in-context learning capabilities of transformers to dynamically adapt to individual preferences. Our approach consists of two phases: (1) an offline phase where we train a single policy model using a history-dependent loss function, and (2) an online phase where the model adapts to user preferences through in-context learning. We demonstrate PPT's effectiveness in a contextual bandit setting, showing that it achieves personalized adaptation superior to existing methods while significantly reducing the computational costs. Our results suggest the potential of in-context learning for scalable and efficient personalization in large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14001 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Personalized Adaptation via In-Context Preference Learning Lau, Allison Choi, Younwoo Balazadeh, Vahid Chidambaram, Keertana Syrgkanis, Vasilis Krishnan, Rahul G. Machine Learning Computation and Language Reinforcement Learning from Human Feedback (RLHF) is widely used to align Language Models (LMs) with human preferences. However, existing approaches often neglect individual user preferences, leading to suboptimal personalization. We present the Preference Pretrained Transformer (PPT), a novel approach for adaptive personalization using online user feedback. PPT leverages the in-context learning capabilities of transformers to dynamically adapt to individual preferences. Our approach consists of two phases: (1) an offline phase where we train a single policy model using a history-dependent loss function, and (2) an online phase where the model adapts to user preferences through in-context learning. We demonstrate PPT's effectiveness in a contextual bandit setting, showing that it achieves personalized adaptation superior to existing methods while significantly reducing the computational costs. Our results suggest the potential of in-context learning for scalable and efficient personalization in large language models. |
| title | Personalized Adaptation via In-Context Preference Learning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2410.14001 |