SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915514518863872 |
|---|---|
| author | Zeng, Xiyu Liang, Siyuan Lu, Liming Zhu, Haotian Liu, Enguang Dang, Jisheng Zhou, Yongbin Pang, Shuchao |
| author_facet | Zeng, Xiyu Liang, Siyuan Lu, Liming Zhu, Haotian Liu, Enguang Dang, Jisheng Zhou, Yongbin Pang, Shuchao |
| contents | As the capabilities of Vision Language Models (VLMs) continue to improve, they are increasingly targeted by jailbreak attacks. Existing defense methods face two major limitations: (1) they struggle to ensure safety without compromising the model's utility; and (2) many defense mechanisms significantly reduce the model's inference efficiency. To address these challenges, we propose SafeSteer, a lightweight, inference-time steering framework that effectively defends against diverse jailbreak attacks without modifying model weights. At the core of SafeSteer is the innovative use of Singular Value Decomposition to construct a low-dimensional "safety subspace." By projecting and reconstructing the raw steering vector into this subspace during inference, SafeSteer adaptively removes harmful generation signals while preserving the model's ability to handle benign inputs. The entire process is executed in a single inference pass, introducing negligible overhead. Extensive experiments show that SafeSteer reduces the attack success rate by over 60% and improves accuracy on normal tasks by 1-2%, without introducing significant inference latency. These results demonstrate that robust and practical jailbreak defense can be achieved through simple, efficient inference-time control. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_21400 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models Zeng, Xiyu Liang, Siyuan Lu, Liming Zhu, Haotian Liu, Enguang Dang, Jisheng Zhou, Yongbin Pang, Shuchao Cryptography and Security As the capabilities of Vision Language Models (VLMs) continue to improve, they are increasingly targeted by jailbreak attacks. Existing defense methods face two major limitations: (1) they struggle to ensure safety without compromising the model's utility; and (2) many defense mechanisms significantly reduce the model's inference efficiency. To address these challenges, we propose SafeSteer, a lightweight, inference-time steering framework that effectively defends against diverse jailbreak attacks without modifying model weights. At the core of SafeSteer is the innovative use of Singular Value Decomposition to construct a low-dimensional "safety subspace." By projecting and reconstructing the raw steering vector into this subspace during inference, SafeSteer adaptively removes harmful generation signals while preserving the model's ability to handle benign inputs. The entire process is executed in a single inference pass, introducing negligible overhead. Extensive experiments show that SafeSteer reduces the attack success rate by over 60% and improves accuracy on normal tasks by 1-2%, without introducing significant inference latency. These results demonstrate that robust and practical jailbreak defense can be achieved through simple, efficient inference-time control. |
| title | SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models |
| topic | Cryptography and Security |
| url | https://arxiv.org/abs/2509.21400 |