SECURE: Benchmarking Large Language Models for Cybersecurity
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909371154300928 |
|---|---|
| author | Bhusal, Dipkamal Alam, Md Tanvirul Nguyen, Le Mahara, Ashim Lightcap, Zachary Frazier, Rodney Fieblinger, Romy Torales, Grace Long Blakely, Benjamin A. Rastogi, Nidhi |
| author_facet | Bhusal, Dipkamal Alam, Md Tanvirul Nguyen, Le Mahara, Ashim Lightcap, Zachary Frazier, Rodney Fieblinger, Romy Torales, Grace Long Blakely, Benjamin A. Rastogi, Nidhi |
| contents | Large Language Models (LLMs) have demonstrated potential in cybersecurity applications but have also caused lower confidence due to problems like hallucinations and a lack of truthfulness. Existing benchmarks provide general evaluations but do not sufficiently address the practical and applied aspects of LLM performance in cybersecurity-specific tasks. To address this gap, we introduce the SECURE (Security Extraction, Understanding \& Reasoning Evaluation), a benchmark designed to assess LLMs performance in realistic cybersecurity scenarios. SECURE includes six datasets focussed on the Industrial Control System sector to evaluate knowledge extraction, understanding, and reasoning based on industry-standard sources. Our study evaluates seven state-of-the-art models on these tasks, providing insights into their strengths and weaknesses in cybersecurity contexts, and offer recommendations for improving LLMs reliability as cyber advisory tools. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_20441 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SECURE: Benchmarking Large Language Models for Cybersecurity Bhusal, Dipkamal Alam, Md Tanvirul Nguyen, Le Mahara, Ashim Lightcap, Zachary Frazier, Rodney Fieblinger, Romy Torales, Grace Long Blakely, Benjamin A. Rastogi, Nidhi Cryptography and Security Artificial Intelligence Human-Computer Interaction Large Language Models (LLMs) have demonstrated potential in cybersecurity applications but have also caused lower confidence due to problems like hallucinations and a lack of truthfulness. Existing benchmarks provide general evaluations but do not sufficiently address the practical and applied aspects of LLM performance in cybersecurity-specific tasks. To address this gap, we introduce the SECURE (Security Extraction, Understanding \& Reasoning Evaluation), a benchmark designed to assess LLMs performance in realistic cybersecurity scenarios. SECURE includes six datasets focussed on the Industrial Control System sector to evaluate knowledge extraction, understanding, and reasoning based on industry-standard sources. Our study evaluates seven state-of-the-art models on these tasks, providing insights into their strengths and weaknesses in cybersecurity contexts, and offer recommendations for improving LLMs reliability as cyber advisory tools. |
| title | SECURE: Benchmarking Large Language Models for Cybersecurity |
| topic | Cryptography and Security Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2405.20441 |