Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.00739 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915552215171072 |
|---|---|
| author | Zhang, Chiyu Cote, Marc-Alexandre Albada, Michael Sankaran, Anush Stokes, Jack W. Wang, Tong Abdi, Amir Blum, William Abdul-Mageed, Muhammad |
| author_facet | Zhang, Chiyu Cote, Marc-Alexandre Albada, Michael Sankaran, Anush Stokes, Jack W. Wang, Tong Abdi, Amir Blum, William Abdul-Mageed, Muhammad |
| contents | Large language model (LLM) agents have shown impressive capabilities in human language comprehension and reasoning, yet their potential in cybersecurity remains underexplored. We introduce DefenderBench, a practical, open-source toolkit for evaluating language agents across offense, defense, and cybersecurity knowledge-based tasks. DefenderBench includes environments for network intrusion, malicious content detection, code vulnerability analysis, and cybersecurity knowledge assessment. It is intentionally designed to be affordable and easily accessible for researchers while providing fair and rigorous assessment. We benchmark several state-of-the-art (SoTA) and popular LLMs, including both open- and closed-weight models, using a standardized agentic framework. Our results show that Claude-3.7-sonnet performs best with a DefenderBench score of 81.65, followed by Claude-3.7-sonnet-think with 78.40, while the best open-weight model, Llama 3.3 70B, is not far behind with a DefenderBench score of 71.81. DefenderBench's modular design allows seamless integration of custom LLMs and tasks, promoting reproducibility and fair comparisons. An anonymized version of DefenderBench is available at https://github.com/microsoft/DefenderBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00739 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments Zhang, Chiyu Cote, Marc-Alexandre Albada, Michael Sankaran, Anush Stokes, Jack W. Wang, Tong Abdi, Amir Blum, William Abdul-Mageed, Muhammad Computation and Language Large language model (LLM) agents have shown impressive capabilities in human language comprehension and reasoning, yet their potential in cybersecurity remains underexplored. We introduce DefenderBench, a practical, open-source toolkit for evaluating language agents across offense, defense, and cybersecurity knowledge-based tasks. DefenderBench includes environments for network intrusion, malicious content detection, code vulnerability analysis, and cybersecurity knowledge assessment. It is intentionally designed to be affordable and easily accessible for researchers while providing fair and rigorous assessment. We benchmark several state-of-the-art (SoTA) and popular LLMs, including both open- and closed-weight models, using a standardized agentic framework. Our results show that Claude-3.7-sonnet performs best with a DefenderBench score of 81.65, followed by Claude-3.7-sonnet-think with 78.40, while the best open-weight model, Llama 3.3 70B, is not far behind with a DefenderBench score of 71.81. DefenderBench's modular design allows seamless integration of custom LLMs and tasks, promoting reproducibility and fair comparisons. An anonymized version of DefenderBench is available at https://github.com/microsoft/DefenderBench. |
| title | DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.00739 |