CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910691283173376 |
|---|---|
| author | Alam, Md Tanvirul Bhusal, Dipkamal Nguyen, Le Rastogi, Nidhi |
| author_facet | Alam, Md Tanvirul Bhusal, Dipkamal Nguyen, Le Rastogi, Nidhi |
| contents | Cyber threat intelligence (CTI) is crucial in today's cybersecurity landscape, providing essential insights to understand and mitigate the ever-evolving cyber threats. The recent rise of Large Language Models (LLMs) have shown potential in this domain, but concerns about their reliability, accuracy, and hallucinations persist. While existing benchmarks provide general evaluations of LLMs, there are no benchmarks that address the practical and applied aspects of CTI-specific tasks. To bridge this gap, we introduce CTIBench, a benchmark designed to assess LLMs' performance in CTI applications. CTIBench includes multiple datasets focused on evaluating knowledge acquired by LLMs in the cyber-threat landscape. Our evaluation of several state-of-the-art models on these tasks provides insights into their strengths and weaknesses in CTI contexts, contributing to a better understanding of LLM capabilities in CTI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07599 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence Alam, Md Tanvirul Bhusal, Dipkamal Nguyen, Le Rastogi, Nidhi Cryptography and Security Artificial Intelligence Cyber threat intelligence (CTI) is crucial in today's cybersecurity landscape, providing essential insights to understand and mitigate the ever-evolving cyber threats. The recent rise of Large Language Models (LLMs) have shown potential in this domain, but concerns about their reliability, accuracy, and hallucinations persist. While existing benchmarks provide general evaluations of LLMs, there are no benchmarks that address the practical and applied aspects of CTI-specific tasks. To bridge this gap, we introduce CTIBench, a benchmark designed to assess LLMs' performance in CTI applications. CTIBench includes multiple datasets focused on evaluating knowledge acquired by LLMs in the cyber-threat landscape. Our evaluation of several state-of-the-art models on these tasks provides insights into their strengths and weaknesses in CTI contexts, contributing to a better understanding of LLM capabilities in CTI. |
| title | CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence |
| topic | Cryptography and Security Artificial Intelligence |
| url | https://arxiv.org/abs/2406.07599 |