BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909939254951936 |
|---|---|
| author | Zhang, Andy K. Ji, Joey Menders, Celeste Dulepet, Riya Qin, Thomas Wang, Ron Y. Wu, Junrong Liao, Kyleen Li, Jiliang Hu, Jinghan Hong, Sara Demilew, Nardos Murgai, Shivatmica Tran, Jason Kacheria, Nishka Ho, Ethan Liu, Denis McLane, Lauren Bruvik, Olivia Han, Dai-Rong Kim, Seungwoo Vyas, Akhil Chen, Cuiyuanxiu Li, Ryan Xu, Weiran Ye, Jonathan Z. Choudhary, Prerit Bhatia, Siddharth M. Sivashankar, Vikram Bao, Yuxuan Song, Dawn Boneh, Dan Ho, Daniel E. Liang, Percy |
| author_facet | Zhang, Andy K. Ji, Joey Menders, Celeste Dulepet, Riya Qin, Thomas Wang, Ron Y. Wu, Junrong Liao, Kyleen Li, Jiliang Hu, Jinghan Hong, Sara Demilew, Nardos Murgai, Shivatmica Tran, Jason Kacheria, Nishka Ho, Ethan Liu, Denis McLane, Lauren Bruvik, Olivia Han, Dai-Rong Kim, Seungwoo Vyas, Akhil Chen, Cuiyuanxiu Li, Ryan Xu, Weiran Ye, Jonathan Z. Choudhary, Prerit Bhatia, Siddharth M. Sivashankar, Vikram Bao, Yuxuan Song, Dawn Boneh, Dan Ho, Daniel E. Liang, Percy |
| contents | AI agents have the potential to significantly alter the cybersecurity landscape. Here, we introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems. Instantiating this framework with BountyBench, we set up 25 systems with complex, real-world codebases. To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a given vulnerability), and Patch (patching a given vulnerability). For Detect, we construct a new success indicator, which is general across vulnerability types and provides localized evaluation. We manually set up the environment for each system, including installing packages, setting up server(s), and hydrating database(s). We add 40 bug bounties, which are vulnerabilities with monetary awards from \$10 to \$30,485, covering 9 of the OWASP Top 10 Risks. To modulate task difficulty, we devise a new strategy based on information to guide detection, interpolating from identifying a zero day to exploiting a given vulnerability. We evaluate 10 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, Qwen3 235B A22B, Llama 4 Maverick, and DeepSeek-R1. Given up to three attempts, the top-performing agents are Codex CLI: o3-high (12.5% on Detect, mapping to \$3,720; 90% on Patch, mapping to \$14,152), Custom Agent: Claude 3.7 Sonnet Thinking (67.5% on Exploit), and Codex CLI: o4-mini (90% on Patch, mapping to \$14,422). Codex CLI: o3-high, Codex CLI: o4-mini, and Claude Code are more capable at defense, achieving higher Patch scores of 90%, 90%, and 87.5%, compared to Exploit scores of 47.5%, 32.5%, and 57.5% respectively; while the custom agents are relatively balanced between offense and defense, achieving Exploit scores of 17.5-67.5% and Patch scores of 25-60%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15216 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems Zhang, Andy K. Ji, Joey Menders, Celeste Dulepet, Riya Qin, Thomas Wang, Ron Y. Wu, Junrong Liao, Kyleen Li, Jiliang Hu, Jinghan Hong, Sara Demilew, Nardos Murgai, Shivatmica Tran, Jason Kacheria, Nishka Ho, Ethan Liu, Denis McLane, Lauren Bruvik, Olivia Han, Dai-Rong Kim, Seungwoo Vyas, Akhil Chen, Cuiyuanxiu Li, Ryan Xu, Weiran Ye, Jonathan Z. Choudhary, Prerit Bhatia, Siddharth M. Sivashankar, Vikram Bao, Yuxuan Song, Dawn Boneh, Dan Ho, Daniel E. Liang, Percy Cryptography and Security Artificial Intelligence Computation and Language Machine Learning AI agents have the potential to significantly alter the cybersecurity landscape. Here, we introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems. Instantiating this framework with BountyBench, we set up 25 systems with complex, real-world codebases. To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a given vulnerability), and Patch (patching a given vulnerability). For Detect, we construct a new success indicator, which is general across vulnerability types and provides localized evaluation. We manually set up the environment for each system, including installing packages, setting up server(s), and hydrating database(s). We add 40 bug bounties, which are vulnerabilities with monetary awards from \$10 to \$30,485, covering 9 of the OWASP Top 10 Risks. To modulate task difficulty, we devise a new strategy based on information to guide detection, interpolating from identifying a zero day to exploiting a given vulnerability. We evaluate 10 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, Qwen3 235B A22B, Llama 4 Maverick, and DeepSeek-R1. Given up to three attempts, the top-performing agents are Codex CLI: o3-high (12.5% on Detect, mapping to \$3,720; 90% on Patch, mapping to \$14,152), Custom Agent: Claude 3.7 Sonnet Thinking (67.5% on Exploit), and Codex CLI: o4-mini (90% on Patch, mapping to \$14,422). Codex CLI: o3-high, Codex CLI: o4-mini, and Claude Code are more capable at defense, achieving higher Patch scores of 90%, 90%, and 87.5%, compared to Exploit scores of 47.5%, 32.5%, and 57.5% respectively; while the custom agents are relatively balanced between offense and defense, achieving Exploit scores of 17.5-67.5% and Patch scores of 25-60%. |
| title | BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems |
| topic | Cryptography and Security Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.15216 |