CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915610381778944 |
|---|---|
| author | Deason, Lauren Bali, Adam Bejean, Ciprian Bolocan, Diana Crnkovich, James Croitoru, Ioana Durai, Krishna Midler, Chase Miron, Calin Molnar, David Moon, Brad Ostarcevic, Bruno Peltea, Alberto Rosenberg, Matt Sandu, Catalin Saputkin, Arthur Shah, Sagar Stan, Daniel Szocs, Ernest Wan, Shengye Whitman, Spencer Krasser, Sven Saxe, Joshua |
| author_facet | Deason, Lauren Bali, Adam Bejean, Ciprian Bolocan, Diana Crnkovich, James Croitoru, Ioana Durai, Krishna Midler, Chase Miron, Calin Molnar, David Moon, Brad Ostarcevic, Bruno Peltea, Alberto Rosenberg, Matt Sandu, Catalin Saputkin, Arthur Shah, Sagar Stan, Daniel Szocs, Ernest Wan, Shengye Whitman, Spencer Krasser, Sven Saxe, Joshua |
| contents | Today's cyber defenders are overwhelmed by a deluge of security alerts, threat intelligence signals, and shifting business context, creating an urgent need for AI systems to enhance operational security work. While Large Language Models (LLMs) have the potential to automate and scale Security Operations Center (SOC) operations, existing evaluations do not fully assess the scenarios most relevant to real-world defenders. This lack of informed evaluation impacts both AI developers and those applying LLMs to SOC automation. Without clear insight into LLM performance in real-world security scenarios, developers lack a north star for development, and users cannot reliably select the most effective models. Meanwhile, malicious actors are using AI to scale cyber attacks, highlighting the need for open source benchmarks to drive adoption and community-driven improvement among defenders and model developers. To address this, we introduce CyberSOCEval, a new suite of open source benchmarks within CyberSecEval 4. CyberSOCEval includes benchmarks tailored to evaluate LLMs in two tasks: Malware Analysis and Threat Intelligence Reasoning--core defensive domains with inadequate coverage in current benchmarks. Our evaluations show that larger, more modern LLMs tend to perform better, confirming the training scaling laws paradigm. We also find that reasoning models leveraging test time scaling do not achieve the same boost as in coding and math, suggesting these models have not been trained to reason about cybersecurity analysis, and pointing to a key opportunity for improvement. Finally, current LLMs are far from saturating our evaluations, showing that CyberSOCEval presents a significant challenge for AI developers to improve cyber defense capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20166 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning Deason, Lauren Bali, Adam Bejean, Ciprian Bolocan, Diana Crnkovich, James Croitoru, Ioana Durai, Krishna Midler, Chase Miron, Calin Molnar, David Moon, Brad Ostarcevic, Bruno Peltea, Alberto Rosenberg, Matt Sandu, Catalin Saputkin, Arthur Shah, Sagar Stan, Daniel Szocs, Ernest Wan, Shengye Whitman, Spencer Krasser, Sven Saxe, Joshua Cryptography and Security Artificial Intelligence Today's cyber defenders are overwhelmed by a deluge of security alerts, threat intelligence signals, and shifting business context, creating an urgent need for AI systems to enhance operational security work. While Large Language Models (LLMs) have the potential to automate and scale Security Operations Center (SOC) operations, existing evaluations do not fully assess the scenarios most relevant to real-world defenders. This lack of informed evaluation impacts both AI developers and those applying LLMs to SOC automation. Without clear insight into LLM performance in real-world security scenarios, developers lack a north star for development, and users cannot reliably select the most effective models. Meanwhile, malicious actors are using AI to scale cyber attacks, highlighting the need for open source benchmarks to drive adoption and community-driven improvement among defenders and model developers. To address this, we introduce CyberSOCEval, a new suite of open source benchmarks within CyberSecEval 4. CyberSOCEval includes benchmarks tailored to evaluate LLMs in two tasks: Malware Analysis and Threat Intelligence Reasoning--core defensive domains with inadequate coverage in current benchmarks. Our evaluations show that larger, more modern LLMs tend to perform better, confirming the training scaling laws paradigm. We also find that reasoning models leveraging test time scaling do not achieve the same boost as in coding and math, suggesting these models have not been trained to reason about cybersecurity analysis, and pointing to a key opportunity for improvement. Finally, current LLMs are far from saturating our evaluations, showing that CyberSOCEval presents a significant challenge for AI developers to improve cyber defense capabilities. |
| title | CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning |
| topic | Cryptography and Security Artificial Intelligence |
| url | https://arxiv.org/abs/2509.20166 |