Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914363930050560 |
|---|---|
| author | Lin, Justin W. Jones, Eliot Krzysztof Jasper, Donovan Julian Ho, Ethan Jun-shen Wu, Anna Yang, Arnold Tianyi Perry, Neil Zou, Andy Fredrikson, Matt Kolter, J. Zico Liang, Percy Boneh, Dan Ho, Daniel E. |
| author_facet | Lin, Justin W. Jones, Eliot Krzysztof Jasper, Donovan Julian Ho, Ethan Jun-shen Wu, Anna Yang, Arnold Tianyi Perry, Neil Zou, Andy Fredrikson, Matt Kolter, J. Zico Liang, Percy Boneh, Dan Ho, Daniel E. |
| contents | We present the first comprehensive evaluation of AI agents against human cybersecurity professionals in a live enterprise environment. We evaluate ten cybersecurity professionals alongside six existing AI agents and ARTEMIS, our new agent scaffold, on a large university network consisting of ~8,000 hosts across 12 subnets. ARTEMIS is a multi-agent framework featuring dynamic prompt generation, arbitrary sub-agents, and automatic vulnerability triaging. In our comparative study, ARTEMIS placed second overall, discovering 9 valid vulnerabilities with an 82% valid submission rate and outperforming 9 of 10 human participants. While existing scaffolds such as Codex and CyAgent underperformed relative to most human participants, ARTEMIS demonstrated technical sophistication and submission quality comparable to the strongest participants. We observe that AI agents offer advantages in systematic enumeration, parallel exploitation, and cost -- certain ARTEMIS variants cost $18/hour versus $60/hour for professional penetration testers. We also identify key capability gaps: AI agents exhibit higher false-positive rates and struggle with GUI-based tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_09882 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing Lin, Justin W. Jones, Eliot Krzysztof Jasper, Donovan Julian Ho, Ethan Jun-shen Wu, Anna Yang, Arnold Tianyi Perry, Neil Zou, Andy Fredrikson, Matt Kolter, J. Zico Liang, Percy Boneh, Dan Ho, Daniel E. Artificial Intelligence Cryptography and Security Computers and Society We present the first comprehensive evaluation of AI agents against human cybersecurity professionals in a live enterprise environment. We evaluate ten cybersecurity professionals alongside six existing AI agents and ARTEMIS, our new agent scaffold, on a large university network consisting of ~8,000 hosts across 12 subnets. ARTEMIS is a multi-agent framework featuring dynamic prompt generation, arbitrary sub-agents, and automatic vulnerability triaging. In our comparative study, ARTEMIS placed second overall, discovering 9 valid vulnerabilities with an 82% valid submission rate and outperforming 9 of 10 human participants. While existing scaffolds such as Codex and CyAgent underperformed relative to most human participants, ARTEMIS demonstrated technical sophistication and submission quality comparable to the strongest participants. We observe that AI agents offer advantages in systematic enumeration, parallel exploitation, and cost -- certain ARTEMIS variants cost $18/hour versus $60/hour for professional penetration testers. We also identify key capability gaps: AI agents exhibit higher false-positive rates and struggle with GUI-based tasks. |
| title | Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing |
| topic | Artificial Intelligence Cryptography and Security Computers and Society |
| url | https://arxiv.org/abs/2512.09882 |