Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.08723 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915929611304960 |
|---|---|
| author | Lee, Chia-Hsuan Zhou, Mingyang Ni, Renkun Cheng, Zelei Dai, Sihui Chakraborty, Supriyo Zhang, Shixiong Sahu, Sambit Campbell, William |
| author_facet | Lee, Chia-Hsuan Zhou, Mingyang Ni, Renkun Cheng, Zelei Dai, Sihui Chakraborty, Supriyo Zhang, Shixiong Sahu, Sambit Campbell, William |
| contents | Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream reasoning gains. We ask: what aspects of a preference pair improve a reasoning model's performance on general reasoning tasks? We investigate two distinct notions of quality delta in preference data: generator-level delta, arising from the differences in capability between models that generate chosen and rejected reasoning traces, and sample-level delta, arising from differences in judged quality differences within an individual preference pair. To study generator-level delta, we vary the generator's scale and model family, and to study sample-level delta, we employ an LLM-as-a-judge to rate the quality of generated traces along multiple reasoning-quality dimensions. We find that increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks and filtering data by sample-level delta can enable more data-efficient training. Our results suggest a twofold recipe for improving reasoning performance through preference optimization: maximize generator-level delta when constructing preference pairs and exploit sample-level delta to select the most informative training examples. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08723 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Decomposing the Delta: What Do Models Actually Learn from Preference Pairs? Lee, Chia-Hsuan Zhou, Mingyang Ni, Renkun Cheng, Zelei Dai, Sihui Chakraborty, Supriyo Zhang, Shixiong Sahu, Sambit Campbell, William Computation and Language Artificial Intelligence Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream reasoning gains. We ask: what aspects of a preference pair improve a reasoning model's performance on general reasoning tasks? We investigate two distinct notions of quality delta in preference data: generator-level delta, arising from the differences in capability between models that generate chosen and rejected reasoning traces, and sample-level delta, arising from differences in judged quality differences within an individual preference pair. To study generator-level delta, we vary the generator's scale and model family, and to study sample-level delta, we employ an LLM-as-a-judge to rate the quality of generated traces along multiple reasoning-quality dimensions. We find that increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks and filtering data by sample-level delta can enable more data-efficient training. Our results suggest a twofold recipe for improving reasoning performance through preference optimization: maximize generator-level delta when constructing preference pairs and exploit sample-level delta to select the most informative training examples. |
| title | Decomposing the Delta: What Do Models Actually Learn from Preference Pairs? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.08723 |