Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.17260 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908785509924864 |
|---|---|
| author | Pollanen, Marco |
| author_facet | Pollanen, Marco |
| contents | Direct Preference Optimization (DPO) is often tuned as if increasing alignment pressure (controlled by $β$) yields progressively "better" behavior. We instead treat $β$ as a control parameter and densely sweep it for three 7B open-weight families under a fixed DPO recipe. In Mistral, capability is sharply non-monotonic: aggregated logic-probe margins become positive only in a narrow band near $β\approx 10^{-2}$ and revert outside it, with boundary points that are seed-sensitive. Across architectures under the same sweep, we observe qualitatively different response modes: sharp reorganization in Mistral, selective changes in Llama, and smooth trade-offs in Qwen. Critically, the DPO preference margin can anticorrelate with reasoning capability (Pearson $r=-0.91$ for Llama logic), so margin-based selection can prefer capability-impaired models. Training path also matters: exposure to high $β$ induces capability losses that persist even after $β$ is reduced (hysteresis). These findings motivate capability-resolved evaluation across the $β$ landscape rather than reliance on margins or aggregate benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17260 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The Viscosity of Logic: Phase Transitions and Hysteresis in DPO Alignment Pollanen, Marco Machine Learning Artificial Intelligence Direct Preference Optimization (DPO) is often tuned as if increasing alignment pressure (controlled by $β$) yields progressively "better" behavior. We instead treat $β$ as a control parameter and densely sweep it for three 7B open-weight families under a fixed DPO recipe. In Mistral, capability is sharply non-monotonic: aggregated logic-probe margins become positive only in a narrow band near $β\approx 10^{-2}$ and revert outside it, with boundary points that are seed-sensitive. Across architectures under the same sweep, we observe qualitatively different response modes: sharp reorganization in Mistral, selective changes in Llama, and smooth trade-offs in Qwen. Critically, the DPO preference margin can anticorrelate with reasoning capability (Pearson $r=-0.91$ for Llama logic), so margin-based selection can prefer capability-impaired models. Training path also matters: exposure to high $β$ induces capability losses that persist even after $β$ is reduced (hysteresis). These findings motivate capability-resolved evaluation across the $β$ landscape rather than reliance on margins or aggregate benchmarks. |
| title | The Viscosity of Logic: Phase Transitions and Hysteresis in DPO Alignment |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.17260 |