CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912017983471616 |
|---|---|
| author | Wan, Shengye Nikolaidis, Cyrus Song, Daniel Molnar, David Crnkovich, James Grace, Jayson Bhatt, Manish Chennabasappa, Sahana Whitman, Spencer Ding, Stephanie Ionescu, Vlad Li, Yue Saxe, Joshua |
| author_facet | Wan, Shengye Nikolaidis, Cyrus Song, Daniel Molnar, David Crnkovich, James Grace, Jayson Bhatt, Manish Chennabasappa, Sahana Whitman, Spencer Ding, Stephanie Ionescu, Vlad Li, Yue Saxe, Joshua |
| contents | We are releasing a new suite of security benchmarks for LLMs, CYBERSECEVAL 3, to continue the conversation on empirically measuring LLM cybersecurity risks and capabilities. CYBERSECEVAL 3 assesses 8 different risks across two broad categories: risk to third parties, and risk to application developers and end users. Compared to previous work, we add new areas focused on offensive security capabilities: automated social engineering, scaling manual offensive cyber operations, and autonomous offensive cyber operations. In this paper we discuss applying these benchmarks to the Llama 3 models and a suite of contemporaneous state-of-the-art LLMs, enabling us to contextualize risks both with and without mitigations in place. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01605 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models Wan, Shengye Nikolaidis, Cyrus Song, Daniel Molnar, David Crnkovich, James Grace, Jayson Bhatt, Manish Chennabasappa, Sahana Whitman, Spencer Ding, Stephanie Ionescu, Vlad Li, Yue Saxe, Joshua Cryptography and Security Machine Learning We are releasing a new suite of security benchmarks for LLMs, CYBERSECEVAL 3, to continue the conversation on empirically measuring LLM cybersecurity risks and capabilities. CYBERSECEVAL 3 assesses 8 different risks across two broad categories: risk to third parties, and risk to application developers and end users. Compared to previous work, we add new areas focused on offensive security capabilities: automated social engineering, scaling manual offensive cyber operations, and autonomous offensive cyber operations. In this paper we discuss applying these benchmarks to the Llama 3 models and a suite of contemporaneous state-of-the-art LLMs, enabling us to contextualize risks both with and without mitigations in place. |
| title | CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models |
| topic | Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2408.01605 |