Cover Image

The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ferrao, Jeremias, van der Lende, Matthijs, Lichkovski, Ilija, Neo, Clement
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

EU-Agent-Bench: Measuring Illegal Behavior of LLM Agents Under EU Law
by: Lichkovski, Ilija, et al.
Published: (2025)

Evaluating Uncertainty in Deep Gaussian Processes
by: van der Lende, Matthijs, et al.
Published: (2025)

Preference Learning for AI Alignment: a Causal Perspective
by: Kobalczyk, Katarzyna, et al.
Published: (2025)

Inference-time Alignment via Sparse Junction Steering
by: Hu, Runyi, et al.
Published: (2026)

DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment
by: Wedgwood, James, et al.
Published: (2026)

Sample Efficient Preference Alignment in LLMs via Active Exploration
by: Mehta, Viraj, et al.
Published: (2023)

Improving Steering Vectors by Targeting Sparse Autoencoder Features
by: Chalnev, Sviatoslav, et al.
Published: (2024)

Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
by: Badrinath, Anirudhan, et al.
Published: (2024)

Causal Language Control in Multilingual Transformers via Sparse Feature Steering
by: Chou, Cheng-Ting, et al.
Published: (2025)

Cross-Modal Content Optimization for Steering Web Agent Preferences
by: Jiang, Tanqiu, et al.
Published: (2025)

Deep Gaussian Process Proximal Policy Optimization
by: van der Lende, Matthijs, et al.
Published: (2025)

Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization
by: Ambadkar, Tanmay, et al.
Published: (2026)

Preference Ranking Optimization for Human Alignment
by: Song, Feifan, et al.
Published: (2023)

Creative Preference Optimization
by: Ismayilzada, Mete, et al.
Published: (2025)

SDQ: Sparse Decomposed Quantization for LLM Inference
by: Jeong, Geonhwa, et al.
Published: (2024)

SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks
by: Christopoulou, Fenia, et al.
Published: (2024)

Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions
by: Neo, Clement, et al.
Published: (2024)

Certified Robustness against Sparse Adversarial Perturbations via Data Localization
by: Pal, Ambar, et al.
Published: (2024)

Multi-Stakeholder LLM Alignment: Decomposing Estimation from Aggregation
by: Zheng, Lulu, et al.
Published: (2026)

Improving Safety Alignment via Balanced Direct Preference Optimization
by: Zhao, Shiji, et al.
Published: (2026)

Unveiling the Power of Sparse Neural Networks for Feature Selection
by: Atashgahi, Zahra, et al.
Published: (2024)

Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders
by: Zhang, Ruikang, et al.
Published: (2026)

Human Preferences for Constructive Interactions in Language Model Alignment
by: Kyrychenko, Yara, et al.
Published: (2025)

HelpSteer2-Preference: Complementing Ratings with Preferences
by: Wang, Zhilin, et al.
Published: (2024)

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization
by: Zhou, Zhanhui, et al.
Published: (2023)

HPS: Hard Preference Sampling for Human Preference Alignment
by: Zou, Xiandong, et al.
Published: (2025)

On the Limitations of Steering in Language Model Alignment
by: Niranjan, Chebrolu, et al.
Published: (2025)

Beyond Preferences in AI Alignment
by: Zhi-Xuan, Tan, et al.
Published: (2024)

CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
by: Cho, Seonglae, et al.
Published: (2025)

Generalized Preference Optimization: A Unified Approach to Offline Alignment
by: Tang, Yunhao, et al.
Published: (2024)

Enhancing Multilingual Counterfactual Generation through Alignment-as-Preference Optimization
by: Wang, Yilong, et al.
Published: (2026)

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
by: Sun, Shangwen, et al.
Published: (2026)

Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
by: Dong, Zhijin
Published: (2025)

Strong Preferences Affect the Robustness of Preference Models and Value Alignment
by: Xu, Ziwei, et al.
Published: (2024)

LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering
by: Wong, Sing Hieng, et al.
Published: (2026)

SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks
by: Kundurthy, Srivatsa, et al.
Published: (2026)

Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?
by: Lee, Chia-Hsuan, et al.
Published: (2026)

YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation
by: Bounhar, Abdelaziz, et al.
Published: (2026)

Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models
by: Wu, Sihao, et al.
Published: (2025)

Accelerated Preference Optimization for Large Language Model Alignment
by: He, Jiafan, et al.
Published: (2024)