Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.20408 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918512888381440 |
|---|---|
| author | Chow, Yinlam Tennenholtz, Guy Yun, Ted Harrison, James Gretton, Arthur Barreto, Andre Dai, Bo |
| author_facet | Chow, Yinlam Tennenholtz, Guy Yun, Ted Harrison, James Gretton, Arthur Barreto, Andre Dai, Bo |
| contents | Reinforcement Learning from Human Feedback (RLHF) effectively aligns Large Language Models (LLMs) with aggregate human preferences but often fails to address the diverse and conflicting needs of individual users. To overcome this issue, we introduce Spectral Souping, a unified framework for efficient, online preference alignment. Our contribution is the discovery of a universal spectral representation within LLMs, which is proven to be highly amenable to model merging. This theoretical insight enables a two-phase methodology: we first learn a basis of specialized policies offline, each focused on a distinct, fine-grained preference dimension. An online adaptation algorithm then efficiently ``soups'' these policies at inference time, either by merging their outputs or parameters, enabling rapid model adaptation without the need for costly online retraining w.r.t. tailored preference rewards. Experiments on online preference alignment benchmarks demonstrate that our method achieves significant performance improvements over existing state-of-the-art approaches, presenting a scalable and computationally efficient solution for dynamically adapting LLMs to individual user preferences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20408 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Spectral Souping: A Unified Framework for Online Preference Alignment Chow, Yinlam Tennenholtz, Guy Yun, Ted Harrison, James Gretton, Arthur Barreto, Andre Dai, Bo Machine Learning Reinforcement Learning from Human Feedback (RLHF) effectively aligns Large Language Models (LLMs) with aggregate human preferences but often fails to address the diverse and conflicting needs of individual users. To overcome this issue, we introduce Spectral Souping, a unified framework for efficient, online preference alignment. Our contribution is the discovery of a universal spectral representation within LLMs, which is proven to be highly amenable to model merging. This theoretical insight enables a two-phase methodology: we first learn a basis of specialized policies offline, each focused on a distinct, fine-grained preference dimension. An online adaptation algorithm then efficiently ``soups'' these policies at inference time, either by merging their outputs or parameters, enabling rapid model adaptation without the need for costly online retraining w.r.t. tailored preference rewards. Experiments on online preference alignment benchmarks demonstrate that our method achieves significant performance improvements over existing state-of-the-art approaches, presenting a scalable and computationally efficient solution for dynamically adapting LLMs to individual user preferences. |
| title | Spectral Souping: A Unified Framework for Online Preference Alignment |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.20408 |