Orchestrating LLMs with Different Personalizations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910514585534464 |
|---|---|
| author | Zhou, Jin Peng Luo, Katie Z Gu, Jingwen Yuan, Jason Weinberger, Kilian Q. Sun, Wen |
| author_facet | Zhou, Jin Peng Luo, Katie Z Gu, Jingwen Yuan, Jason Weinberger, Kilian Q. Sun, Wen |
| contents | This paper presents a novel approach to aligning large language models (LLMs) with individual human preferences, sometimes referred to as Reinforcement Learning from \textit{Personalized} Human Feedback (RLPHF). Given stated preferences along multiple dimensions, such as helpfulness, conciseness, or humor, the goal is to create an LLM without re-training that best adheres to this specification. Starting from specialized expert LLMs, each trained for one such particular preference dimension, we propose a black-box method that merges their outputs on a per-token level. We train a lightweight Preference Control Model (PCM) that dynamically translates the preference description and current context into next-token prediction weights. By combining the expert models' outputs at the token level, our approach dynamically generates text that optimizes the given preference. Empirical tests show that our method matches or surpasses existing preference merging techniques, providing a scalable, efficient alternative to fine-tuning LLMs for individual personalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_04181 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Orchestrating LLMs with Different Personalizations Zhou, Jin Peng Luo, Katie Z Gu, Jingwen Yuan, Jason Weinberger, Kilian Q. Sun, Wen Artificial Intelligence Computation and Language This paper presents a novel approach to aligning large language models (LLMs) with individual human preferences, sometimes referred to as Reinforcement Learning from \textit{Personalized} Human Feedback (RLPHF). Given stated preferences along multiple dimensions, such as helpfulness, conciseness, or humor, the goal is to create an LLM without re-training that best adheres to this specification. Starting from specialized expert LLMs, each trained for one such particular preference dimension, we propose a black-box method that merges their outputs on a per-token level. We train a lightweight Preference Control Model (PCM) that dynamically translates the preference description and current context into next-token prediction weights. By combining the expert models' outputs at the token level, our approach dynamically generates text that optimizes the given preference. Empirical tests show that our method matches or surpasses existing preference merging techniques, providing a scalable, efficient alternative to fine-tuning LLMs for individual personalization. |
| title | Orchestrating LLMs with Different Personalizations |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2407.04181 |