Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911196105408512 |
|---|---|
| author | Yin, Qingyu Leong, Chak Tou Yang, Linyi Huang, Wenxuan Li, Wenjie Wang, Xiting Yoon, Jaehong YunXing XingYu Gu, Jinjin |
| author_facet | Yin, Qingyu Leong, Chak Tou Yang, Linyi Huang, Wenxuan Li, Wenjie Wang, Xiting Yoon, Jaehong YunXing XingYu Gu, Jinjin |
| contents | Large reasoning models (LRMs) with multi-step reasoning capabilities have shown remarkable problem-solving abilities, yet they exhibit concerning safety vulnerabilities that remain poorly understood. In this work, we investigate why safety alignment fails in reasoning models through a mechanistic interpretability lens. Using a linear probing approach to trace refusal intentions across token positions, we discover a striking phenomenon termed as \textbf{refusal cliff}: many poorly-aligned reasoning models correctly identify harmful prompts and maintain strong refusal intentions during their thinking process, but experience a sharp drop in refusal scores at the final tokens before output generation. This suggests that these models are not inherently unsafe; rather, their refusal intentions are systematically suppressed. Through causal intervention analysis, we identify a sparse set of attention heads that negatively contribute to refusal behavior. Ablating just 3\% of these heads can reduce attack success rates below 10\%. Building on these mechanistic insights, we propose \textbf{Cliff-as-a-Judge}, a novel data selection method that identifies training examples exhibiting the largest refusal cliff to efficiently repair reasoning models' safety alignment. This approach achieves comparable safety improvements using only 1.7\% of the vanilla safety training data, demonstrating a less-is-more effect in safety alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06036 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning? Yin, Qingyu Leong, Chak Tou Yang, Linyi Huang, Wenxuan Li, Wenjie Wang, Xiting Yoon, Jaehong YunXing XingYu Gu, Jinjin Artificial Intelligence Cryptography and Security Large reasoning models (LRMs) with multi-step reasoning capabilities have shown remarkable problem-solving abilities, yet they exhibit concerning safety vulnerabilities that remain poorly understood. In this work, we investigate why safety alignment fails in reasoning models through a mechanistic interpretability lens. Using a linear probing approach to trace refusal intentions across token positions, we discover a striking phenomenon termed as \textbf{refusal cliff}: many poorly-aligned reasoning models correctly identify harmful prompts and maintain strong refusal intentions during their thinking process, but experience a sharp drop in refusal scores at the final tokens before output generation. This suggests that these models are not inherently unsafe; rather, their refusal intentions are systematically suppressed. Through causal intervention analysis, we identify a sparse set of attention heads that negatively contribute to refusal behavior. Ablating just 3\% of these heads can reduce attack success rates below 10\%. Building on these mechanistic insights, we propose \textbf{Cliff-as-a-Judge}, a novel data selection method that identifies training examples exhibiting the largest refusal cliff to efficiently repair reasoning models' safety alignment. This approach achieves comparable safety improvements using only 1.7\% of the vanilla safety training data, demonstrating a less-is-more effect in safety alignment. |
| title | Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning? |
| topic | Artificial Intelligence Cryptography and Security |
| url | https://arxiv.org/abs/2510.06036 |