EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909637819760640 |
|---|---|
| author | Abramovich, Talor Udeshi, Meet Shao, Minghao Lieret, Kilian Xi, Haoran Milner, Kimberly Jancheska, Sofija Yang, John Jimenez, Carlos E. Khorrami, Farshad Krishnamurthy, Prashanth Dolan-Gavitt, Brendan Shafique, Muhammad Narasimhan, Karthik Karri, Ramesh Press, Ofir |
| author_facet | Abramovich, Talor Udeshi, Meet Shao, Minghao Lieret, Kilian Xi, Haoran Milner, Kimberly Jancheska, Sofija Yang, John Jimenez, Carlos E. Khorrami, Farshad Krishnamurthy, Prashanth Dolan-Gavitt, Brendan Shafique, Muhammad Narasimhan, Karthik Karri, Ramesh Press, Ofir |
| contents | Although language model (LM) agents have demonstrated increased performance in multiple domains, including coding and web-browsing, their success in cybersecurity has been limited. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities, focusing on interactive terminal programs. These novel Interactive Agent Tools enable LM agents, for the first time, to run interactive utilities, such as a debugger and a server connection tool, which are essential for solving these challenges. Empirical analysis on 390 CTF challenges across four benchmarks demonstrate that these new tools and interfaces substantially improve our agent's performance, achieving state-of-the-art results on NYU CTF, Intercode-CTF, and CyBench. Finally, we analyze data leakage, developing new methods to quantify it and identifying a new phenomenon we term soliloquizing, where the model self-generates hallucinated observations without interacting with the environment. Our code and development dataset are available at https://github.com/SWE-agent/SWE-agent/tree/v0.7 and https://github.com/NYU-LLM-CTF/NYU_CTF_Bench/tree/main/development respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_16165 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities Abramovich, Talor Udeshi, Meet Shao, Minghao Lieret, Kilian Xi, Haoran Milner, Kimberly Jancheska, Sofija Yang, John Jimenez, Carlos E. Khorrami, Farshad Krishnamurthy, Prashanth Dolan-Gavitt, Brendan Shafique, Muhammad Narasimhan, Karthik Karri, Ramesh Press, Ofir Artificial Intelligence Although language model (LM) agents have demonstrated increased performance in multiple domains, including coding and web-browsing, their success in cybersecurity has been limited. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities, focusing on interactive terminal programs. These novel Interactive Agent Tools enable LM agents, for the first time, to run interactive utilities, such as a debugger and a server connection tool, which are essential for solving these challenges. Empirical analysis on 390 CTF challenges across four benchmarks demonstrate that these new tools and interfaces substantially improve our agent's performance, achieving state-of-the-art results on NYU CTF, Intercode-CTF, and CyBench. Finally, we analyze data leakage, developing new methods to quantify it and identifying a new phenomenon we term soliloquizing, where the model self-generates hallucinated observations without interacting with the environment. Our code and development dataset are available at https://github.com/SWE-agent/SWE-agent/tree/v0.7 and https://github.com/NYU-LLM-CTF/NYU_CTF_Bench/tree/main/development respectively. |
| title | EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2409.16165 |