Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.24068 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908813898022912 |
|---|---|
| author | McKenzie, Ian R. Hollinsworth, Oskar J. Tseng, Tom Davies, Xander Casper, Stephen Tucker, Aaron D. Kirk, Robert Gleave, Adam |
| author_facet | McKenzie, Ian R. Hollinsworth, Oskar J. Tseng, Tom Davies, Xander Casper, Stephen Tucker, Aaron D. Kirk, Robert Gleave, Adam |
| contents | Frontier AI developers are relying on layers of safeguards to protect against catastrophic misuse of AI systems. Anthropic and OpenAI guard their latest Opus 4 model and GPT-5 models using such defense pipelines, and other frontier developers including Google DeepMind pledge to soon deploy similar defenses. However, the security of such pipelines is unclear, with limited prior work evaluating or attacking these pipelines. We address this gap by developing and red-teaming an open-source defense pipeline. First, we find that a novel few-shot-prompted input and output classifier outperforms state-of-the-art open-weight safeguard model ShieldGemma across three attacks and two datasets, reducing the attack success rate (ASR) to 0% on the catastrophic misuse dataset ClearHarm. Second, we introduce a STaged AttaCK (STACK) procedure that achieves 71% ASR on ClearHarm in a black-box attack against the few-shot-prompted classifier pipeline. Finally, we also evaluate STACK in a transfer setting, achieving 33% ASR, providing initial evidence that it is feasible to design attacks with no access to the target pipeline. We conclude by suggesting specific mitigations that developers could use to thwart staged attacks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_24068 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | STACK: Adversarial Attacks on LLM Safeguard Pipelines McKenzie, Ian R. Hollinsworth, Oskar J. Tseng, Tom Davies, Xander Casper, Stephen Tucker, Aaron D. Kirk, Robert Gleave, Adam Computation and Language Artificial Intelligence Frontier AI developers are relying on layers of safeguards to protect against catastrophic misuse of AI systems. Anthropic and OpenAI guard their latest Opus 4 model and GPT-5 models using such defense pipelines, and other frontier developers including Google DeepMind pledge to soon deploy similar defenses. However, the security of such pipelines is unclear, with limited prior work evaluating or attacking these pipelines. We address this gap by developing and red-teaming an open-source defense pipeline. First, we find that a novel few-shot-prompted input and output classifier outperforms state-of-the-art open-weight safeguard model ShieldGemma across three attacks and two datasets, reducing the attack success rate (ASR) to 0% on the catastrophic misuse dataset ClearHarm. Second, we introduce a STaged AttaCK (STACK) procedure that achieves 71% ASR on ClearHarm in a black-box attack against the few-shot-prompted classifier pipeline. Finally, we also evaluate STACK in a transfer setting, achieving 33% ASR, providing initial evidence that it is feasible to design attacks with no access to the target pipeline. We conclude by suggesting specific mitigations that developers could use to thwart staged attacks. |
| title | STACK: Adversarial Attacks on LLM Safeguard Pipelines |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.24068 |