Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.11296 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909684787576832 |
|---|---|
| author | O'Brien, Kyle Majercak, David Fernandes, Xavier Edgar, Richard Bullwinkel, Blake Chen, Jingya Nori, Harsha Carignan, Dean Horvitz, Eric Poursabzi-Sangdeh, Forough |
| author_facet | O'Brien, Kyle Majercak, David Fernandes, Xavier Edgar, Richard Bullwinkel, Blake Chen, Jingya Nori, Harsha Carignan, Dean Horvitz, Eric Poursabzi-Sangdeh, Forough |
| contents | Responsible deployment of language models requires mechanisms for refusing unsafe prompts while preserving model performance. While most approaches modify model weights through additional training, we explore an alternative: steering model activations at inference time via amplifying sparse autoencoder (SAE) features that mediate refusal. This work uncovers a fundamental tension between SAE steering-based safety improvements and general model capabilities. While feature steering successfully improves robustness against both single-turn and challenging multi-turn jailbreak attempts, we discover that this comes at a previously underexplored cost -- systematic degradation of performance across multiple benchmark tasks, even on safe inputs with no apparent connection to refusal behavior. This suggests that features mediating refusal may be more deeply entangled with general language model capabilities than previously understood. Our findings reveal important open questions about the nature of safety-relevant features in language models and the feasibility of isolating them for targeted intervention. While SAE-based steering shows promise as a flexible approach to enhancing language model safety, our results highlight the critical need to understand and address the mechanisms behind these capability tradeoffs before such techniques can be practically deployed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_11296 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Steering Language Model Refusal with Sparse Autoencoders O'Brien, Kyle Majercak, David Fernandes, Xavier Edgar, Richard Bullwinkel, Blake Chen, Jingya Nori, Harsha Carignan, Dean Horvitz, Eric Poursabzi-Sangdeh, Forough Machine Learning Responsible deployment of language models requires mechanisms for refusing unsafe prompts while preserving model performance. While most approaches modify model weights through additional training, we explore an alternative: steering model activations at inference time via amplifying sparse autoencoder (SAE) features that mediate refusal. This work uncovers a fundamental tension between SAE steering-based safety improvements and general model capabilities. While feature steering successfully improves robustness against both single-turn and challenging multi-turn jailbreak attempts, we discover that this comes at a previously underexplored cost -- systematic degradation of performance across multiple benchmark tasks, even on safe inputs with no apparent connection to refusal behavior. This suggests that features mediating refusal may be more deeply entangled with general language model capabilities than previously understood. Our findings reveal important open questions about the nature of safety-relevant features in language models and the feasibility of isolating them for targeted intervention. While SAE-based steering shows promise as a flexible approach to enhancing language model safety, our results highlight the critical need to understand and address the mechanisms behind these capability tradeoffs before such techniques can be practically deployed. |
| title | Steering Language Model Refusal with Sparse Autoencoders |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.11296 |