Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908469376843776 |
|---|---|
| author | Zou, Andy Lin, Maxwell Jones, Eliot Nowak, Micha Dziemian, Mateusz Winter, Nick Grattan, Alexander Nathanael, Valent Croft, Ayla Davies, Xander Patel, Jai Kirk, Robert Burnikell, Nate Gal, Yarin Hendrycks, Dan Kolter, J. Zico Fredrikson, Matt |
| author_facet | Zou, Andy Lin, Maxwell Jones, Eliot Nowak, Micha Dziemian, Mateusz Winter, Nick Grattan, Alexander Nathanael, Valent Croft, Ayla Davies, Xander Patel, Jai Kirk, Robert Burnikell, Nate Gal, Yarin Hendrycks, Dan Kolter, J. Zico Fredrikson, Matt |
| contents | Recent advances have enabled LLM-powered AI agents to autonomously execute complex tasks by combining language model reasoning with tools, memory, and web access. But can these systems be trusted to follow deployment policies in realistic environments, especially under attack? To investigate, we ran the largest public red-teaming competition to date, targeting 22 frontier AI agents across 44 realistic deployment scenarios. Participants submitted 1.8 million prompt-injection attacks, with over 60,000 successfully eliciting policy violations such as unauthorized data access, illicit financial actions, and regulatory noncompliance. We use these results to build the Agent Red Teaming (ART) benchmark - a curated set of high-impact attacks - and evaluate it across 19 state-of-the-art models. Nearly all agents exhibit policy violations for most behaviors within 10-100 queries, with high attack transferability across models and tasks. Importantly, we find limited correlation between agent robustness and model size, capability, or inference-time compute, suggesting that additional defenses are needed against adversarial misuse. Our findings highlight critical and persistent vulnerabilities in today's AI agents. By releasing the ART benchmark and accompanying evaluation framework, we aim to support more rigorous security assessment and drive progress toward safer agent deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_20526 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition Zou, Andy Lin, Maxwell Jones, Eliot Nowak, Micha Dziemian, Mateusz Winter, Nick Grattan, Alexander Nathanael, Valent Croft, Ayla Davies, Xander Patel, Jai Kirk, Robert Burnikell, Nate Gal, Yarin Hendrycks, Dan Kolter, J. Zico Fredrikson, Matt Artificial Intelligence Computation and Language Computers and Society Recent advances have enabled LLM-powered AI agents to autonomously execute complex tasks by combining language model reasoning with tools, memory, and web access. But can these systems be trusted to follow deployment policies in realistic environments, especially under attack? To investigate, we ran the largest public red-teaming competition to date, targeting 22 frontier AI agents across 44 realistic deployment scenarios. Participants submitted 1.8 million prompt-injection attacks, with over 60,000 successfully eliciting policy violations such as unauthorized data access, illicit financial actions, and regulatory noncompliance. We use these results to build the Agent Red Teaming (ART) benchmark - a curated set of high-impact attacks - and evaluate it across 19 state-of-the-art models. Nearly all agents exhibit policy violations for most behaviors within 10-100 queries, with high attack transferability across models and tasks. Importantly, we find limited correlation between agent robustness and model size, capability, or inference-time compute, suggesting that additional defenses are needed against adversarial misuse. Our findings highlight critical and persistent vulnerabilities in today's AI agents. By releasing the ART benchmark and accompanying evaluation framework, we aim to support more rigorous security assessment and drive progress toward safer agent deployment. |
| title | Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition |
| topic | Artificial Intelligence Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2507.20526 |