Detoxifying LLMs via Representation Erasure-Based Preference Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912929514782720 |
|---|---|
| author | Sepahvand, Nazanin Mohammadi Triantafillou, Eleni Larochelle, Hugo Precup, Doina Roy, Daniel M. Dziugaite, Gintare Karolina |
| author_facet | Sepahvand, Nazanin Mohammadi Triantafillou, Eleni Larochelle, Hugo Precup, Doina Roy, Daniel M. Dziugaite, Gintare Karolina |
| contents | Large language models (LLMs) trained on webscale data can produce toxic outputs, raising concerns for safe deployment. Prior defenses, based on applications of DPO, NPO, and similar algorithms, reduce the likelihood of harmful continuations, but not robustly so: they are vulnerable to adversarial prompting and easily undone by fine-tuning-based relearning attacks. Indeed, research has shown that these edits to the model are superficial: linear probing reveals that harmful "directions" remain present in representations. To address this, we propose Representation Erasure-based Preference Optimization (REPO), reformulating detoxification as a token-level preference problem. Using a novel objective with preference data, we force the representations of toxic continuations to converge toward their benign counterparts. Our mechanistic analysis reveals that this granular approach is critical: unlike baselines, REPO induces deep, localized edits to toxicity-encoding neurons while preserving general model utility. Exhaustive evaluations show that REPO achieves state-of-the-art robustness, stopping sophisticated threats-including relearning attacks and enhanced GCG jailbreaks-where existing representation- and output-based methods fail. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_23391 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Detoxifying LLMs via Representation Erasure-Based Preference Optimization Sepahvand, Nazanin Mohammadi Triantafillou, Eleni Larochelle, Hugo Precup, Doina Roy, Daniel M. Dziugaite, Gintare Karolina Machine Learning Large language models (LLMs) trained on webscale data can produce toxic outputs, raising concerns for safe deployment. Prior defenses, based on applications of DPO, NPO, and similar algorithms, reduce the likelihood of harmful continuations, but not robustly so: they are vulnerable to adversarial prompting and easily undone by fine-tuning-based relearning attacks. Indeed, research has shown that these edits to the model are superficial: linear probing reveals that harmful "directions" remain present in representations. To address this, we propose Representation Erasure-based Preference Optimization (REPO), reformulating detoxification as a token-level preference problem. Using a novel objective with preference data, we force the representations of toxic continuations to converge toward their benign counterparts. Our mechanistic analysis reveals that this granular approach is critical: unlike baselines, REPO induces deep, localized edits to toxicity-encoding neurons while preserving general model utility. Exhaustive evaluations show that REPO achieves state-of-the-art robustness, stopping sophisticated threats-including relearning attacks and enhanced GCG jailbreaks-where existing representation- and output-based methods fail. |
| title | Detoxifying LLMs via Representation Erasure-Based Preference Optimization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.23391 |