CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916569333891072 |
|---|---|
| author | Yu, Zhengmin Zeng, Jiutian Chen, Siyi Xu, Wenhan Xu, Dandan Liu, Xiangyu Ying, Zonghao Wang, Nan Zhang, Yuan Yang, Min |
| author_facet | Yu, Zhengmin Zeng, Jiutian Chen, Siyi Xu, Wenhan Xu, Dandan Liu, Xiangyu Ying, Zonghao Wang, Nan Zhang, Yuan Yang, Min |
| contents | Over the past year, there has been a notable rise in the use of large language models (LLMs) for academic research and industrial practices within the cybersecurity field. However, it remains a lack of comprehensive and publicly accessible benchmarks to evaluate the performance of LLMs on cybersecurity tasks. To address this gap, we introduce CS-Eval, a publicly accessible, comprehensive and bilingual LLM benchmark specifically designed for cybersecurity. CS-Eval synthesizes the research hotspots from academia and practical applications from industry, curating a diverse set of high-quality questions across 42 categories within cybersecurity, systematically organized into three cognitive levels: knowledge, ability, and application. Through an extensive evaluation of a wide range of LLMs using CS-Eval, we have uncovered valuable insights. For instance, while GPT-4 generally excels overall, other models may outperform it in certain specific subcategories. Additionally, by conducting evaluations over several months, we observed significant improvements in many LLMs' abilities to solve cybersecurity tasks. The benchmarks are now publicly available at https://github.com/CS-EVAL/CS-Eval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16239 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity Yu, Zhengmin Zeng, Jiutian Chen, Siyi Xu, Wenhan Xu, Dandan Liu, Xiangyu Ying, Zonghao Wang, Nan Zhang, Yuan Yang, Min Cryptography and Security Over the past year, there has been a notable rise in the use of large language models (LLMs) for academic research and industrial practices within the cybersecurity field. However, it remains a lack of comprehensive and publicly accessible benchmarks to evaluate the performance of LLMs on cybersecurity tasks. To address this gap, we introduce CS-Eval, a publicly accessible, comprehensive and bilingual LLM benchmark specifically designed for cybersecurity. CS-Eval synthesizes the research hotspots from academia and practical applications from industry, curating a diverse set of high-quality questions across 42 categories within cybersecurity, systematically organized into three cognitive levels: knowledge, ability, and application. Through an extensive evaluation of a wide range of LLMs using CS-Eval, we have uncovered valuable insights. For instance, while GPT-4 generally excels overall, other models may outperform it in certain specific subcategories. Additionally, by conducting evaluations over several months, we observed significant improvements in many LLMs' abilities to solve cybersecurity tasks. The benchmarks are now publicly available at https://github.com/CS-EVAL/CS-Eval. |
| title | CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity |
| topic | Cryptography and Security |
| url | https://arxiv.org/abs/2411.16239 |