CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915300147986432 |
|---|---|
| author | Shao, Minghao Xi, Haoran Rani, Nanda Udeshi, Meet Putrevu, Venkata Sai Charan Milner, Kimberly Dolan-Gavitt, Brendan Shukla, Sandeep Kumar Krishnamurthy, Prashanth Khorrami, Farshad Karri, Ramesh Shafique, Muhammad |
| author_facet | Shao, Minghao Xi, Haoran Rani, Nanda Udeshi, Meet Putrevu, Venkata Sai Charan Milner, Kimberly Dolan-Gavitt, Brendan Shukla, Sandeep Kumar Krishnamurthy, Prashanth Khorrami, Farshad Karri, Ramesh Shafique, Muhammad |
| contents | Large Language Model (LLM) agents can automate cybersecurity tasks and can adapt to the evolving cybersecurity landscape without re-engineering. While LLM agents have demonstrated cybersecurity capabilities on Capture-The-Flag (CTF) competitions, they have two key limitations: accessing latest cybersecurity expertise beyond training data, and integrating new knowledge into complex task planning. Knowledge-based approaches that incorporate technical understanding into the task-solving automation can tackle these limitations. We present CRAKEN, a knowledge-based LLM agent framework that improves cybersecurity capability through three core mechanisms: contextual decomposition of task-critical information, iterative self-reflected knowledge retrieval, and knowledge-hint injection that transforms insights into adaptive attack strategies. Comprehensive evaluations with different configurations show CRAKEN's effectiveness in multi-stage vulnerability detection and exploitation compared to previous approaches. Our extensible architecture establishes new methodologies for embedding new security knowledge into LLM-driven cybersecurity agentic systems. With a knowledge database of CTF writeups, CRAKEN obtained an accuracy of 22% on NYU CTF Bench, outperforming prior works by 3% and achieving state-of-the-art results. On evaluation of MITRE ATT&CK techniques, CRAKEN solves 25-30% more techniques than prior work, demonstrating improved cybersecurity capabilities via knowledge-based execution. We make our framework open source to public https://github.com/NYU-LLM-CTF/nyuctf_agents_craken. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17107 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution Shao, Minghao Xi, Haoran Rani, Nanda Udeshi, Meet Putrevu, Venkata Sai Charan Milner, Kimberly Dolan-Gavitt, Brendan Shukla, Sandeep Kumar Krishnamurthy, Prashanth Khorrami, Farshad Karri, Ramesh Shafique, Muhammad Cryptography and Security Artificial Intelligence Machine Learning Multiagent Systems Large Language Model (LLM) agents can automate cybersecurity tasks and can adapt to the evolving cybersecurity landscape without re-engineering. While LLM agents have demonstrated cybersecurity capabilities on Capture-The-Flag (CTF) competitions, they have two key limitations: accessing latest cybersecurity expertise beyond training data, and integrating new knowledge into complex task planning. Knowledge-based approaches that incorporate technical understanding into the task-solving automation can tackle these limitations. We present CRAKEN, a knowledge-based LLM agent framework that improves cybersecurity capability through three core mechanisms: contextual decomposition of task-critical information, iterative self-reflected knowledge retrieval, and knowledge-hint injection that transforms insights into adaptive attack strategies. Comprehensive evaluations with different configurations show CRAKEN's effectiveness in multi-stage vulnerability detection and exploitation compared to previous approaches. Our extensible architecture establishes new methodologies for embedding new security knowledge into LLM-driven cybersecurity agentic systems. With a knowledge database of CTF writeups, CRAKEN obtained an accuracy of 22% on NYU CTF Bench, outperforming prior works by 3% and achieving state-of-the-art results. On evaluation of MITRE ATT&CK techniques, CRAKEN solves 25-30% more techniques than prior work, demonstrating improved cybersecurity capabilities via knowledge-based execution. We make our framework open source to public https://github.com/NYU-LLM-CTF/nyuctf_agents_craken. |
| title | CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution |
| topic | Cryptography and Security Artificial Intelligence Machine Learning Multiagent Systems |
| url | https://arxiv.org/abs/2505.17107 |