Biothreat Benchmark Generation Framework for Evaluating Frontier AI Models II: Benchmark Generation Process
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912755586433024 |
|---|---|
| author | Ackerman, Gary Kallenborn, Zachary Wetzel, Anna Peterson, Hayley LaTourette, Jenna Shoemaker, Olivia Behlendorf, Brandon Almakki, Sheriff Clifford, Doug Sheinbaum, Noah |
| author_facet | Ackerman, Gary Kallenborn, Zachary Wetzel, Anna Peterson, Hayley LaTourette, Jenna Shoemaker, Olivia Behlendorf, Brandon Almakki, Sheriff Clifford, Doug Sheinbaum, Noah |
| contents | The potential for rapidly-evolving frontier artificial intelligence (AI) models, especially large language models (LLMs), to facilitate bioterrorism or access to biological weapons has generated significant policy, academic, and public concern. Both model developers and policymakers seek to quantify and mitigate any risk, with an important element of such efforts being the development of model benchmarks that can assess the biosecurity risk posed by a particular model. This paper, the second in a series of three, describes the second component of a novel Biothreat Benchmark Generation (BBG) framework: the generation of the Bacterial Biothreat Benchmark (B3) dataset. The development process involved three complementary approaches: 1) web-based prompt generation, 2) red teaming, and 3) mining existing benchmark corpora, to generate over 7,000 potential benchmarks linked to the Task-Query Architecture that was developed during the first component of the project. A process of de-duplication, followed by an assessment of uplift diagnosticity, and general quality control measures, reduced the candidates to a set of 1,010 final benchmarks. This procedure ensured that these benchmarks are a) diagnostic in terms of providing uplift; b) directly relevant to biosecurity threats; and c) are aligned with a larger biosecurity architecture permitting nuanced analysis at different levels of analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_08451 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Biothreat Benchmark Generation Framework for Evaluating Frontier AI Models II: Benchmark Generation Process Ackerman, Gary Kallenborn, Zachary Wetzel, Anna Peterson, Hayley LaTourette, Jenna Shoemaker, Olivia Behlendorf, Brandon Almakki, Sheriff Clifford, Doug Sheinbaum, Noah Machine Learning Artificial Intelligence Computers and Society Emerging Technologies The potential for rapidly-evolving frontier artificial intelligence (AI) models, especially large language models (LLMs), to facilitate bioterrorism or access to biological weapons has generated significant policy, academic, and public concern. Both model developers and policymakers seek to quantify and mitigate any risk, with an important element of such efforts being the development of model benchmarks that can assess the biosecurity risk posed by a particular model. This paper, the second in a series of three, describes the second component of a novel Biothreat Benchmark Generation (BBG) framework: the generation of the Bacterial Biothreat Benchmark (B3) dataset. The development process involved three complementary approaches: 1) web-based prompt generation, 2) red teaming, and 3) mining existing benchmark corpora, to generate over 7,000 potential benchmarks linked to the Task-Query Architecture that was developed during the first component of the project. A process of de-duplication, followed by an assessment of uplift diagnosticity, and general quality control measures, reduced the candidates to a set of 1,010 final benchmarks. This procedure ensured that these benchmarks are a) diagnostic in terms of providing uplift; b) directly relevant to biosecurity threats; and c) are aligned with a larger biosecurity architecture permitting nuanced analysis at different levels of analysis. |
| title | Biothreat Benchmark Generation Framework for Evaluating Frontier AI Models II: Benchmark Generation Process |
| topic | Machine Learning Artificial Intelligence Computers and Society Emerging Technologies |
| url | https://arxiv.org/abs/2512.08451 |