When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916046615609344 |
|---|---|
| author | Le, Khoi Cao, Tri Nguyen, Phong Nguyen, Cong-Duy Luu, Anh Tuan Chunyan, Miao Ng, See-Kiong Nguyen, Thong |
| author_facet | Le, Khoi Cao, Tri Nguyen, Phong Nguyen, Cong-Duy Luu, Anh Tuan Chunyan, Miao Ng, See-Kiong Nguyen, Thong |
| contents | Weak-to-strong (W2S) generalization is a promising framework for scalable oversight, yet existing evaluations often test students under matched train-test distributions. Therefore, we study W2S preference learning under zero-shot distribution shift and find that strong students trained on weak preference labels can appear successful in-distribution while failing to transfer across preference datasets. We provide evidence for a representational failure mode in which weak-supervised fine-tuning can pull the strong model toward source-domain features instead of maintaining broadly transferable preference representations. To mitigate this, we propose Representation Anchoring (Anchor), a simple yet effective regularizer that constrains excessive drift from the pretrained strong model's representation space during fine-tuning, while still allowing task-relevant adaptation. Across preference domains, datasets, and model families, Anchor consistently improves out-of-distribution transfer while maintaining competitive in-distribution performance. Together, our evaluation protocol, transfer-aware metrics, and method expose hidden brittleness in current W2S reward modeling and provide a practical path toward more robust preference transfer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25629 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift Le, Khoi Cao, Tri Nguyen, Phong Nguyen, Cong-Duy Luu, Anh Tuan Chunyan, Miao Ng, See-Kiong Nguyen, Thong Computation and Language Machine Learning Weak-to-strong (W2S) generalization is a promising framework for scalable oversight, yet existing evaluations often test students under matched train-test distributions. Therefore, we study W2S preference learning under zero-shot distribution shift and find that strong students trained on weak preference labels can appear successful in-distribution while failing to transfer across preference datasets. We provide evidence for a representational failure mode in which weak-supervised fine-tuning can pull the strong model toward source-domain features instead of maintaining broadly transferable preference representations. To mitigate this, we propose Representation Anchoring (Anchor), a simple yet effective regularizer that constrains excessive drift from the pretrained strong model's representation space during fine-tuning, while still allowing task-relevant adaptation. Across preference domains, datasets, and model families, Anchor consistently improves out-of-distribution transfer while maintaining competitive in-distribution performance. Together, our evaluation protocol, transfer-aware metrics, and method expose hidden brittleness in current W2S reward modeling and provide a practical path toward more robust preference transfer. |
| title | When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2605.25629 |