ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913078477586432 |
|---|---|
| author | Wu, Yiran Velazco, Mauricio Zhao, Andrew Luján, Manuel Raúl Meléndez Movva, Srisuma Roy, Yogesh K Nguyen, Quang Rodriguez, Roberto Wu, Qingyun Albada, Michael Kiseleva, Julia Mudgerikar, Anand |
| author_facet | Wu, Yiran Velazco, Mauricio Zhao, Andrew Luján, Manuel Raúl Meléndez Movva, Srisuma Roy, Yogesh K Nguyen, Quang Rodriguez, Roberto Wu, Qingyun Albada, Michael Kiseleva, Julia Mudgerikar, Anand |
| contents | We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research. The code is available at https://github.com/microsoft/SecRL |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_14201 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation Wu, Yiran Velazco, Mauricio Zhao, Andrew Luján, Manuel Raúl Meléndez Movva, Srisuma Roy, Yogesh K Nguyen, Quang Rodriguez, Roberto Wu, Qingyun Albada, Michael Kiseleva, Julia Mudgerikar, Anand Cryptography and Security Artificial Intelligence Computation and Language We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research. The code is available at https://github.com/microsoft/SecRL |
| title | ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation |
| topic | Cryptography and Security Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2507.14201 |