Drift: Decoding-time Personalized Alignments with Implicit User Preferences
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916725765701632 |
|---|---|
| author | Kim, Minbeom Lee, Kang-il Joo, Seongho Lee, Hwaran Thonet, Thibaut Jung, Kyomin |
| author_facet | Kim, Minbeom Lee, Kang-il Joo, Seongho Lee, Hwaran Thonet, Thibaut Jung, Kyomin |
| contents | Personalized alignments for individual users have been a long-standing goal in large language models (LLMs). We introduce Drift, a novel framework that personalizes LLMs at decoding time with implicit user preferences. Traditional Reinforcement Learning from Human Feedback (RLHF) requires thousands of annotated examples and expensive gradient updates. In contrast, Drift personalizes LLMs in a training-free manner, using only a few dozen examples to steer a frozen model through efficient preference modeling. Our approach models user preferences as a composition of predefined, interpretable attributes and aligns them at decoding time to enable personalized generation. Experiments on both a synthetic persona dataset (Perspective) and a real human-annotated dataset (PRISM) demonstrate that Drift significantly outperforms RLHF baselines while using only 50-100 examples. Our results and analysis show that Drift is both computationally efficient and interpretable. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14289 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Drift: Decoding-time Personalized Alignments with Implicit User Preferences Kim, Minbeom Lee, Kang-il Joo, Seongho Lee, Hwaran Thonet, Thibaut Jung, Kyomin Computation and Language Personalized alignments for individual users have been a long-standing goal in large language models (LLMs). We introduce Drift, a novel framework that personalizes LLMs at decoding time with implicit user preferences. Traditional Reinforcement Learning from Human Feedback (RLHF) requires thousands of annotated examples and expensive gradient updates. In contrast, Drift personalizes LLMs in a training-free manner, using only a few dozen examples to steer a frozen model through efficient preference modeling. Our approach models user preferences as a composition of predefined, interpretable attributes and aligns them at decoding time to enable personalized generation. Experiments on both a synthetic persona dataset (Perspective) and a real human-annotated dataset (PRISM) demonstrate that Drift significantly outperforms RLHF baselines while using only 50-100 examples. Our results and analysis show that Drift is both computationally efficient and interpretable. |
| title | Drift: Decoding-time Personalized Alignments with Implicit User Preferences |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.14289 |