Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908603427848192 |
|---|---|
| author | Balassone, Francesco Mayoral-Vilches, Víctor Rass, Stefan Pinzger, Martin Perrone, Gaetano Romano, Simon Pietro Schartner, Peter |
| author_facet | Balassone, Francesco Mayoral-Vilches, Víctor Rass, Stefan Pinzger, Martin Perrone, Gaetano Romano, Simon Pietro Schartner, Peter |
| contents | We empirically evaluate whether AI systems are more effective at attacking or defending in cybersecurity. Using CAI (Cybersecurity AI)'s parallel execution framework, we deployed autonomous agents in 23 Attack/Defense CTF battlegrounds. Statistical analysis reveals defensive agents achieve 54.3% unconstrained patching success versus 28.3% offensive initial access (p=0.0193), but this advantage disappears under operational constraints: when defense requires maintaining availability (23.9%) and preventing all intrusions (15.2%), no significant difference exists (p>0.05). Exploratory taxonomy analysis suggests potential patterns in vulnerability exploitation, though limited sample sizes preclude definitive conclusions. This study provides the first controlled empirical evidence challenging claims of AI attacker advantage, demonstrating that defensive effectiveness critically depends on success criteria, a nuance absent from conceptual analyses but essential for deployment. These findings underscore the urgency for defenders to adopt open-source Cybersecurity AI frameworks to maintain security equilibrium against accelerating offensive automation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_17521 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs Balassone, Francesco Mayoral-Vilches, Víctor Rass, Stefan Pinzger, Martin Perrone, Gaetano Romano, Simon Pietro Schartner, Peter Cryptography and Security We empirically evaluate whether AI systems are more effective at attacking or defending in cybersecurity. Using CAI (Cybersecurity AI)'s parallel execution framework, we deployed autonomous agents in 23 Attack/Defense CTF battlegrounds. Statistical analysis reveals defensive agents achieve 54.3% unconstrained patching success versus 28.3% offensive initial access (p=0.0193), but this advantage disappears under operational constraints: when defense requires maintaining availability (23.9%) and preventing all intrusions (15.2%), no significant difference exists (p>0.05). Exploratory taxonomy analysis suggests potential patterns in vulnerability exploitation, though limited sample sizes preclude definitive conclusions. This study provides the first controlled empirical evidence challenging claims of AI attacker advantage, demonstrating that defensive effectiveness critically depends on success criteria, a nuance absent from conceptual analyses but essential for deployment. These findings underscore the urgency for defenders to adopt open-source Cybersecurity AI frameworks to maintain security equilibrium against accelerating offensive automation. |
| title | Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs |
| topic | Cryptography and Security |
| url | https://arxiv.org/abs/2510.17521 |