Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.08812 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910144254705664 |
|---|---|
| author | Lee, Yoonho Williams, Jonathan Marklund, Henrik Sharma, Archit Mitchell, Eric Singh, Anikait Finn, Chelsea |
| author_facet | Lee, Yoonho Williams, Jonathan Marklund, Henrik Sharma, Archit Mitchell, Eric Singh, Anikait Finn, Chelsea |
| contents | Reward models trained on aggregate preferences often fail to capture individual users' values, but existing adaptation methods such as fine-tuning or long-context conditioning are too costly for real-time personalization. We propose Hypothesis Reweighting (HyRe), which enables real-time personalization by reweighting ensemble members using just 1-5 labeled examples from the target user or domain. Our method builds on the empirical observation that when different heads capture different valid interpretations of preference data, reweighting them can substantially outperform uniform averaging. HyRe trains a single network with multiple prediction heads that capture different valid interpretations of preference data, then uses a Bayesian update to upweight the heads that best match the target user's preferences. This requires only a single forward pass with negligible (<1%) computational overhead, making it practical for inference-time personalization. We evaluate HyRe across diverse target preference distributions. With as few as five preference pairs per target distribution, HyRe surpasses state-of-the-art reward models on RewardBench at 2B and 8B scale and improves reward model accuracy by 20% across 32 personalization tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08812 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Test-Time Alignment via Hypothesis Reweighting Lee, Yoonho Williams, Jonathan Marklund, Henrik Sharma, Archit Mitchell, Eric Singh, Anikait Finn, Chelsea Machine Learning Reward models trained on aggregate preferences often fail to capture individual users' values, but existing adaptation methods such as fine-tuning or long-context conditioning are too costly for real-time personalization. We propose Hypothesis Reweighting (HyRe), which enables real-time personalization by reweighting ensemble members using just 1-5 labeled examples from the target user or domain. Our method builds on the empirical observation that when different heads capture different valid interpretations of preference data, reweighting them can substantially outperform uniform averaging. HyRe trains a single network with multiple prediction heads that capture different valid interpretations of preference data, then uses a Bayesian update to upweight the heads that best match the target user's preferences. This requires only a single forward pass with negligible (<1%) computational overhead, making it practical for inference-time personalization. We evaluate HyRe across diverse target preference distributions. With as few as five preference pairs per target distribution, HyRe surpasses state-of-the-art reward models on RewardBench at 2B and 8B scale and improves reward model accuracy by 20% across 32 personalization tasks. |
| title | Test-Time Alignment via Hypothesis Reweighting |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2412.08812 |