Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.01166 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910178511683584 |
|---|---|
| author | Stelling, Lily Murray, Malcolm Galizzi, Bruno Schaffelder, Max Campos, Siméon Papadatos, Henry |
| author_facet | Stelling, Lily Murray, Malcolm Galizzi, Bruno Schaffelder, Max Campos, Siméon Papadatos, Henry |
| contents | Following the AI Seoul Summit in 2024, twelve AI companies published frontier AI safety frameworks (Frameworks) outlining their approaches to managing catastrophic risks from advanced AI systems. Emerging legislation increasingly treats these Frameworks as external accountability mechanisms, incorporating them into reporting requirements. But what do the Frameworks actually commit each company to do? This study assesses 12 Frameworks, using 65 weighted criteria, across four dimensions: risk identification, risk analysis \& evaluation, risk treatment, and risk governance. Our criteria adapt established risk management principles from other high-risk industries (e.g. aviation, nuclear power) to the frontier AI context, following Campos et al. (2025). Overall scores range from 34% (Anthropic) to 8% (Cohere), with a median of 18%. Many aspects are missing or under-specified. These low scores may be natural given the nascency of AI risk management compared to industries with decades of practice. Nonetheless, current Frameworks are limited as accountability functions, with vague commitments that make it difficult to predict company decisions, assess whether planned responses are adequate, or determine whether commitments have been kept. Still, higher scores appear feasible within current constraints: a company adopting all leading practices currently adopted across their peers would score 54%, which is triple the current median. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_01166 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating AI Providers' Frontier Safety Frameworks Stelling, Lily Murray, Malcolm Galizzi, Bruno Schaffelder, Max Campos, Siméon Papadatos, Henry Computers and Society Following the AI Seoul Summit in 2024, twelve AI companies published frontier AI safety frameworks (Frameworks) outlining their approaches to managing catastrophic risks from advanced AI systems. Emerging legislation increasingly treats these Frameworks as external accountability mechanisms, incorporating them into reporting requirements. But what do the Frameworks actually commit each company to do? This study assesses 12 Frameworks, using 65 weighted criteria, across four dimensions: risk identification, risk analysis \& evaluation, risk treatment, and risk governance. Our criteria adapt established risk management principles from other high-risk industries (e.g. aviation, nuclear power) to the frontier AI context, following Campos et al. (2025). Overall scores range from 34% (Anthropic) to 8% (Cohere), with a median of 18%. Many aspects are missing or under-specified. These low scores may be natural given the nascency of AI risk management compared to industries with decades of practice. Nonetheless, current Frameworks are limited as accountability functions, with vague commitments that make it difficult to predict company decisions, assess whether planned responses are adequate, or determine whether commitments have been kept. Still, higher scores appear feasible within current constraints: a company adopting all leading practices currently adopted across their peers would score 54%, which is triple the current median. |
| title | Evaluating AI Providers' Frontier Safety Frameworks |
| topic | Computers and Society |
| url | https://arxiv.org/abs/2512.01166 |