Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.05162 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914987531829248 |
|---|---|
| author | Wei, Boyi Huang, Kaixuan Huang, Yangsibo Xie, Tinghao Qi, Xiangyu Xia, Mengzhou Mittal, Prateek Wang, Mengdi Henderson, Peter |
| author_facet | Wei, Boyi Huang, Kaixuan Huang, Yangsibo Xie, Tinghao Qi, Xiangyu Xia, Mengzhou Mittal, Prateek Wang, Mengdi Henderson, Peter |
| contents | Large language models (LLMs) show inherent brittleness in their safety mechanisms, as evidenced by their susceptibility to jailbreaking and even non-malicious fine-tuning. This study explores this brittleness of safety alignment by leveraging pruning and low-rank modifications. We develop methods to identify critical regions that are vital for safety guardrails, and that are disentangled from utility-relevant regions at both the neuron and rank levels. Surprisingly, the isolated regions we find are sparse, comprising about $3\%$ at the parameter level and $2.5\%$ at the rank level. Removing these regions compromises safety without significantly impacting utility, corroborating the inherent brittleness of the model's safety mechanisms. Moreover, we show that LLMs remain vulnerable to low-cost fine-tuning attacks even when modifications to the safety-critical regions are restricted. These findings underscore the urgent need for more robust safety strategies in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05162 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications Wei, Boyi Huang, Kaixuan Huang, Yangsibo Xie, Tinghao Qi, Xiangyu Xia, Mengzhou Mittal, Prateek Wang, Mengdi Henderson, Peter Machine Learning Artificial Intelligence Computation and Language Large language models (LLMs) show inherent brittleness in their safety mechanisms, as evidenced by their susceptibility to jailbreaking and even non-malicious fine-tuning. This study explores this brittleness of safety alignment by leveraging pruning and low-rank modifications. We develop methods to identify critical regions that are vital for safety guardrails, and that are disentangled from utility-relevant regions at both the neuron and rank levels. Surprisingly, the isolated regions we find are sparse, comprising about $3\%$ at the parameter level and $2.5\%$ at the rank level. Removing these regions compromises safety without significantly impacting utility, corroborating the inherent brittleness of the model's safety mechanisms. Moreover, we show that LLMs remain vulnerable to low-cost fine-tuning attacks even when modifications to the safety-critical regions are restricted. These findings underscore the urgent need for more robust safety strategies in LLMs. |
| title | Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2402.05162 |