Cover Image

How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yang, Yushi, Sondej, Filip, Mayne, Harry, Lee, Andrew, Mahdi, Adam
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Collapse of Irrelevant Representations (CIR) Ensures Robust and Non-Disruptive LLM Unlearning
by: Sondej, Filip, et al.
Published: (2025)

Can sparse autoencoders be used to decompose and interpret steering vectors?
by: Mayne, Harry, et al.
Published: (2024)

Robust LLM Unlearning with MUDMAN: Meta-Unlearning with Disruption Masking And Normalization
by: Sondej, Filip, et al.
Published: (2025)

LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations
by: Mayne, Harry, et al.
Published: (2025)

Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
by: Chen, Jianhui, et al.
Published: (2024)

Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Models
by: Beniwal, Himanshu, et al.
Published: (2026)

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity
by: Lee, Andrew, et al.
Published: (2024)

Unsupervised Learning Approaches for Identifying ICU Patient Subgroups: Do Results Generalise?
by: Mayne, Harry, et al.
Published: (2024)

Negation Neglect: When models fail to learn negations in training
by: Mayne, Harry, et al.
Published: (2026)

Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
by: Gupta, Taneesh, et al.
Published: (2024)

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
by: Zhang, Zhengze, et al.
Published: (2025)

Bootstrapping Language Models with DPO Implicit Rewards
by: Chen, Changyu, et al.
Published: (2024)

It Takes Two: Your GRPO Is Secretly DPO
by: Wu, Yihong, et al.
Published: (2025)

DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models
by: Jung, Sunghee, et al.
Published: (2025)

Neuron-Level Knowledge Attribution in Large Language Models
by: Yu, Zeping, et al.
Published: (2023)

SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization
by: He, Chaoyue, et al.
Published: (2026)

Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
by: Yan, Lecheng, et al.
Published: (2026)

Robust Multi-Objective Preference Alignment with Online DPO
by: Gupta, Raghav, et al.
Published: (2025)

Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
by: Shi, Ruizhe, et al.
Published: (2025)

Provably Robust DPO: Aligning Language Models with Noisy Feedback
by: Chowdhury, Sayak Ray, et al.
Published: (2024)

M-QUEST -- Meme Question-Understanding Evaluation on Semantics and Toxicity
by: De Giorgis, Stefano, et al.
Published: (2026)

DOTResize: Reducing LLM Width via Discrete Optimal Transport-based Neuron Merging
by: Verma, Neha, et al.
Published: (2025)

Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
by: Xiao, Teng, et al.
Published: (2024)

Agentic Reinforcement Learning for Search is Unsafe
by: Yang, Yushi, et al.
Published: (2025)

How Does Quantization Affect Multilingual LLMs?
by: Marchisio, Kelly, et al.
Published: (2024)

AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
by: Chen, Shaolong, et al.
Published: (2026)

GIFT: Group-Relative Implicit Fine-Tuning Integrates GRPO with DPO and UNA
by: Wang, Zhichao
Published: (2025)

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
by: Wu, Junkang, et al.
Published: (2024)

Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning
by: Deng, Yihe, et al.
Published: (2024)

InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization
by: Wang, Yunan, et al.
Published: (2025)

Tracking the Feature Dynamics in LLM Training: A Mechanistic Study
by: Xu, Yang, et al.
Published: (2024)

How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretability
by: Im, Shawn, et al.
Published: (2026)

DPO Meets PPO: Reinforced Token Optimization for RLHF
by: Zhong, Han, et al.
Published: (2024)

MixDPO: Modeling Preference Strength for Pluralistic Alignment
by: Imai, Saki, et al.
Published: (2026)

Aligning Language Models with Clinical Expertise: DPO for Heart Failure Nursing Documentation in Critical Care
by: Fan, Junyi, et al.
Published: (2025)

Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond
by: Wen, Liang, et al.
Published: (2025)

Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
by: Lai, Xin, et al.
Published: (2024)

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
by: Zhou, Zhenglin, et al.
Published: (2025)

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
by: Wang, Bo, et al.
Published: (2025)

NeuronScope: A Multi-Agent Framework for Explaining Polysemantic Neurons in Language Models
by: Liu, Weiqi, et al.
Published: (2026)