Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.02297 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915830140239872 |
|---|---|
| author | Lau, Nancy Sloot, Louis Raj, Jyoutir Boscardin, Giuseppe Marco Harris, Evan Bowman, Dylan Brajkovski, Mario Chawla, Jaideep Zhao, Dan |
| author_facet | Lau, Nancy Sloot, Louis Raj, Jyoutir Boscardin, Giuseppe Marco Harris, Evan Bowman, Dylan Brajkovski, Mario Chawla, Jaideep Zhao, Dan |
| contents | Large language models (LLMs) are increasingly being deployed as software engineering agents that autonomously contribute to repositories. A major benefit these agents present is their ability to find and patch security vulnerabilities in the codebases they oversee. To estimate the capability of agents in this domain, we introduce ZeroDayBench, a benchmark where LLM agents find and patch 22 novel critical vulnerabilities in open-source codebases. We focus our efforts on three popular frontier agentic LLMs: GPT-5.2, Claude Sonnet 4.5, and Grok 4.1. We find that frontier LLMs are not yet capable of autonomously solving our tasks and observe some behavioral patterns that suggest how these models can be improved in the domain of proactive cyberdefense. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02297 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense Lau, Nancy Sloot, Louis Raj, Jyoutir Boscardin, Giuseppe Marco Harris, Evan Bowman, Dylan Brajkovski, Mario Chawla, Jaideep Zhao, Dan Cryptography and Security Artificial Intelligence Large language models (LLMs) are increasingly being deployed as software engineering agents that autonomously contribute to repositories. A major benefit these agents present is their ability to find and patch security vulnerabilities in the codebases they oversee. To estimate the capability of agents in this domain, we introduce ZeroDayBench, a benchmark where LLM agents find and patch 22 novel critical vulnerabilities in open-source codebases. We focus our efforts on three popular frontier agentic LLMs: GPT-5.2, Claude Sonnet 4.5, and Grok 4.1. We find that frontier LLMs are not yet capable of autonomously solving our tasks and observe some behavioral patterns that suggest how these models can be improved in the domain of proactive cyberdefense. |
| title | ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense |
| topic | Cryptography and Security Artificial Intelligence |
| url | https://arxiv.org/abs/2603.02297 |