Automated Penetration Testing with LLM Agents and Classical Planning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915670870982656 |
|---|---|
| author | Wang, Lingzhi Shi, Xinyi Li, Ziyu Jiang, Yi Tan, Shiyu Jiang, Yuhao Cheng, Junjie Chen, Wenyuan Shen, Xiangmin LI, Zhenyuan Chen, Yan |
| author_facet | Wang, Lingzhi Shi, Xinyi Li, Ziyu Jiang, Yi Tan, Shiyu Jiang, Yuhao Cheng, Junjie Chen, Wenyuan Shen, Xiangmin LI, Zhenyuan Chen, Yan |
| contents | While penetration testing plays a vital role in cybersecurity, achieving fully automated, hands-off-the-keyboard execution remains a significant research challenge. In this paper, we introduce the "Planner-Executor-Perceptor (PEP)" design paradigm and use it to systematically review existing work and identify the key challenges in this area. We also evaluate existing penetration testing systems, with a particular focus on the use of Large Language Model (LLM) agents for this task. The results show that the out-of-the-box Claude Code and Sonnet 4.5 exhibit superior penetration capabilities observed to date, substantially outperforming all prior systems. However, a detailed analysis of their testing processes reveals specific strengths and limitations; notably, LLM agents struggle with maintaining coherent long-horizon plans, performing complex reasoning, and effectively utilizing specialized tools. These limitations significantly constrain its overall capability, efficiency, and stability. To address these limitations, we propose CHECKMATE, a framework that integrates enhanced classical planning with LLM agents, providing an external, structured "brain" that mitigates the inherent weaknesses of LLM agents. Our evaluation shows that CHECKMATE outperforms the state-of-the-art system (Claude Code) in penetration capability, improving benchmark success rates by over 20%. In addition, it delivers substantially greater stability, cutting both time and monetary costs by more than 50%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_11143 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Automated Penetration Testing with LLM Agents and Classical Planning Wang, Lingzhi Shi, Xinyi Li, Ziyu Jiang, Yi Tan, Shiyu Jiang, Yuhao Cheng, Junjie Chen, Wenyuan Shen, Xiangmin LI, Zhenyuan Chen, Yan Cryptography and Security While penetration testing plays a vital role in cybersecurity, achieving fully automated, hands-off-the-keyboard execution remains a significant research challenge. In this paper, we introduce the "Planner-Executor-Perceptor (PEP)" design paradigm and use it to systematically review existing work and identify the key challenges in this area. We also evaluate existing penetration testing systems, with a particular focus on the use of Large Language Model (LLM) agents for this task. The results show that the out-of-the-box Claude Code and Sonnet 4.5 exhibit superior penetration capabilities observed to date, substantially outperforming all prior systems. However, a detailed analysis of their testing processes reveals specific strengths and limitations; notably, LLM agents struggle with maintaining coherent long-horizon plans, performing complex reasoning, and effectively utilizing specialized tools. These limitations significantly constrain its overall capability, efficiency, and stability. To address these limitations, we propose CHECKMATE, a framework that integrates enhanced classical planning with LLM agents, providing an external, structured "brain" that mitigates the inherent weaknesses of LLM agents. Our evaluation shows that CHECKMATE outperforms the state-of-the-art system (Claude Code) in penetration capability, improving benchmark success rates by over 20%. In addition, it delivers substantially greater stability, cutting both time and monetary costs by more than 50%. |
| title | Automated Penetration Testing with LLM Agents and Classical Planning |
| topic | Cryptography and Security |
| url | https://arxiv.org/abs/2512.11143 |