Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.16823 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911278025408512 |
|---|---|
| author | Kim, Joseph Potluri, Saahith |
| author_facet | Kim, Joseph Potluri, Saahith |
| contents | Evaluating and measuring AI Safety Level (ASL) threats are crucial for guiding stakeholders to implement safeguards that keep risks within acceptable limits. ASL-3+ models present a unique risk in their ability to uplift novice non-state actors, especially in the realm of biosecurity. Existing evaluation metrics, such as LAB-Bench, BioLP-bench, and WMDP, can reliably assess model uplift and domain knowledge. However, metrics that better contextualize "real-world risks" are needed to inform the safety case for LLMs, along with scalable, open-ended metrics to keep pace with their rapid advancements. To address both gaps, we introduce MOCET, an interpretable and doubly-scalable metric (automatable and open-ended) that can quantify real-world risks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_16823 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Monte Carlo Expected Threat (MOCET) Scoring Kim, Joseph Potluri, Saahith Machine Learning Artificial Intelligence Human-Computer Interaction 68T01, 65C05, 92C42 I.2.6; K.4.1; J.3; G.3 Evaluating and measuring AI Safety Level (ASL) threats are crucial for guiding stakeholders to implement safeguards that keep risks within acceptable limits. ASL-3+ models present a unique risk in their ability to uplift novice non-state actors, especially in the realm of biosecurity. Existing evaluation metrics, such as LAB-Bench, BioLP-bench, and WMDP, can reliably assess model uplift and domain knowledge. However, metrics that better contextualize "real-world risks" are needed to inform the safety case for LLMs, along with scalable, open-ended metrics to keep pace with their rapid advancements. To address both gaps, we introduce MOCET, an interpretable and doubly-scalable metric (automatable and open-ended) that can quantify real-world risks. |
| title | Monte Carlo Expected Threat (MOCET) Scoring |
| topic | Machine Learning Artificial Intelligence Human-Computer Interaction 68T01, 65C05, 92C42 I.2.6; K.4.1; J.3; G.3 |
| url | https://arxiv.org/abs/2511.16823 |