Hacking CTFs with Plain Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912143224340480 |
|---|---|
| author | Turtayev, Rustem Petrov, Artem Volkov, Dmitrii Volk, Denis |
| author_facet | Turtayev, Rustem Petrov, Artem Volkov, Dmitrii Volk, Denis |
| contents | We saturate a high-school-level hacking benchmark with plain LLM agent design. Concretely, we obtain 95% performance on InterCode-CTF, a popular offensive security benchmark, using prompting, tool use, and multiple attempts. This beats prior work by Phuong et al. 2024 (29%) and Abramovich et al. 2024 (72%).
Our results suggest that current LLMs have surpassed the high school level in offensive cybersecurity. Their hacking capabilities remain underelicited: our ReAct&Plan prompting strategy solves many challenges in 1-2 turns without complex engineering or advanced harnessing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02776 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hacking CTFs with Plain Agents Turtayev, Rustem Petrov, Artem Volkov, Dmitrii Volk, Denis Cryptography and Security Artificial Intelligence We saturate a high-school-level hacking benchmark with plain LLM agent design. Concretely, we obtain 95% performance on InterCode-CTF, a popular offensive security benchmark, using prompting, tool use, and multiple attempts. This beats prior work by Phuong et al. 2024 (29%) and Abramovich et al. 2024 (72%). Our results suggest that current LLMs have surpassed the high school level in offensive cybersecurity. Their hacking capabilities remain underelicited: our ReAct&Plan prompting strategy solves many challenges in 1-2 turns without complex engineering or advanced harnessing. |
| title | Hacking CTFs with Plain Agents |
| topic | Cryptography and Security Artificial Intelligence |
| url | https://arxiv.org/abs/2412.02776 |