SPIRIT: Patching Speech Language Models against Jailbreak Attacks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909848835194880 |
|---|---|
| author | Djanibekov, Amirbek Mukhituly, Nurdaulet Inui, Kentaro Aldarmaki, Hanan Lukas, Nils |
| author_facet | Djanibekov, Amirbek Mukhituly, Nurdaulet Inui, Kentaro Aldarmaki, Hanan Lukas, Nils |
| contents | Speech Language Models (SLMs) enable natural interactions via spoken instructions, which more effectively capture user intent by detecting nuances in speech. The richer speech signal introduces new security risks compared to text-based models, as adversaries can better bypass safety mechanisms by injecting imperceptible noise to speech. We analyze adversarial attacks and find that SLMs are substantially more vulnerable to jailbreak attacks, which can achieve a perfect 100% attack success rate in some instances. To improve security, we propose post-hoc patching defenses used to intervene during inference by modifying the SLM's activations that improve robustness up to 99% with (i) negligible impact on utility and (ii) without any re-training. We conduct ablation studies to maximize the efficacy of our defenses and improve the utility/security trade-off, validated with large-scale benchmarks unique to SLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13541 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SPIRIT: Patching Speech Language Models against Jailbreak Attacks Djanibekov, Amirbek Mukhituly, Nurdaulet Inui, Kentaro Aldarmaki, Hanan Lukas, Nils Audio and Speech Processing Machine Learning Speech Language Models (SLMs) enable natural interactions via spoken instructions, which more effectively capture user intent by detecting nuances in speech. The richer speech signal introduces new security risks compared to text-based models, as adversaries can better bypass safety mechanisms by injecting imperceptible noise to speech. We analyze adversarial attacks and find that SLMs are substantially more vulnerable to jailbreak attacks, which can achieve a perfect 100% attack success rate in some instances. To improve security, we propose post-hoc patching defenses used to intervene during inference by modifying the SLM's activations that improve robustness up to 99% with (i) negligible impact on utility and (ii) without any re-training. We conduct ablation studies to maximize the efficacy of our defenses and improve the utility/security trade-off, validated with large-scale benchmarks unique to SLMs. |
| title | SPIRIT: Patching Speech Language Models against Jailbreak Attacks |
| topic | Audio and Speech Processing Machine Learning |
| url | https://arxiv.org/abs/2505.13541 |