Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2606.02530 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911742392532992 |
|---|---|
| author | Li, Hao An, Jingkun Song, Zijun Zhu, Pengyu Li, Rui Wang, Hao Feng, Wendi Liu, Yesheng Li, Lijun Yao, Jin-Ge Sha, Lei |
| author_facet | Li, Hao An, Jingkun Song, Zijun Zhu, Pengyu Li, Rui Wang, Hao Feng, Wendi Liu, Yesheng Li, Lijun Yao, Jin-Ge Sha, Lei |
| contents | Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax. Existing methods mitigate this by balancing dual objectives, which heavily rely on massive general-purpose data or auxiliary reward models.
In this paper, we argue that, because safety features are inherently sparse within the output distribution, alignment requires localized modifications rather than global trade-offs. To this end, we propose SafeSteer, which performs on-policy distillation confined to safety tokens. First, we construct a safety teacher via activation steering. Based on this teacher, we develop a safety token selection algorithm. Consequently, SafeSteer restricts the reverse KL penalty to these tokens during training to preserve general capabilities.
Experimental results across diverse models show that our SafeSteer achieves a superior trade-off between safety and general capability compared with existing methods, attaining strong safety performance on seven safety benchmarks with only minimal degradation on five general capability benchmarks. Notably, SafeSteer requires only 100 harmful samples without using any general-purpose data, less than 1% of what previous baselines used, considerably reducing alignment cost. More details are on our project page at https://anjingkun.github.io/SafeSteer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_02530 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment Li, Hao An, Jingkun Song, Zijun Zhu, Pengyu Li, Rui Wang, Hao Feng, Wendi Liu, Yesheng Li, Lijun Yao, Jin-Ge Sha, Lei Artificial Intelligence Computation and Language Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax. Existing methods mitigate this by balancing dual objectives, which heavily rely on massive general-purpose data or auxiliary reward models. In this paper, we argue that, because safety features are inherently sparse within the output distribution, alignment requires localized modifications rather than global trade-offs. To this end, we propose SafeSteer, which performs on-policy distillation confined to safety tokens. First, we construct a safety teacher via activation steering. Based on this teacher, we develop a safety token selection algorithm. Consequently, SafeSteer restricts the reverse KL penalty to these tokens during training to preserve general capabilities. Experimental results across diverse models show that our SafeSteer achieves a superior trade-off between safety and general capability compared with existing methods, attaining strong safety performance on seven safety benchmarks with only minimal degradation on five general capability benchmarks. Notably, SafeSteer requires only 100 harmful samples without using any general-purpose data, less than 1% of what previous baselines used, considerably reducing alignment cost. More details are on our project page at https://anjingkun.github.io/SafeSteer. |
| title | SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2606.02530 |