Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.12934 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911295869026304 |
|---|---|
| author | Ferrao, Jeremias van der Lende, Matthijs Lichkovski, Ilija Neo, Clement |
| author_facet | Ferrao, Jeremias van der Lende, Matthijs Lichkovski, Ilija Neo, Clement |
| contents | Prevailing alignment methods induce opaque parameter changes, obscuring what models truly learn. To address this, we introduce Feature Steering with Reinforcement Learning (FSRL), a framework that trains a lightweight adapter to steer model behavior by modulating interpretable sparse features. First, we theoretically demonstrate that this mechanism is expressive enough to approximate the behavioral shifts of post-training processes. We then apply FSRL to preference optimization and perform a causal analysis of the learned policy. Our analysis reveals a crucial insight: the model learns to reward stylistic presentation as a proxy for quality, disproportionately relying on features related to style and formatting over those tied to alignment concepts like honesty. By effectively optimizing the preference objective, FSRL serves as a transparent proxy for observing the alignment process. Overall, FSRL offers an interpretable control interface and a practical way to diagnose how preference optimization pressures manifest at the feature level. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12934 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features Ferrao, Jeremias van der Lende, Matthijs Lichkovski, Ilija Neo, Clement Artificial Intelligence Prevailing alignment methods induce opaque parameter changes, obscuring what models truly learn. To address this, we introduce Feature Steering with Reinforcement Learning (FSRL), a framework that trains a lightweight adapter to steer model behavior by modulating interpretable sparse features. First, we theoretically demonstrate that this mechanism is expressive enough to approximate the behavioral shifts of post-training processes. We then apply FSRL to preference optimization and perform a causal analysis of the learned policy. Our analysis reveals a crucial insight: the model learns to reward stylistic presentation as a proxy for quality, disproportionately relying on features related to style and formatting over those tied to alignment concepts like honesty. By effectively optimizing the preference objective, FSRL serves as a transparent proxy for observing the alignment process. Overall, FSRL offers an interpretable control interface and a practical way to diagnose how preference optimization pressures manifest at the feature level. |
| title | The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.12934 |