Open-World Evaluations for Measuring Frontier AI Capabilities
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917514520297472 |
|---|---|
| author | Kapoor, Sayash Kirgis, Peter Schwartz, Andrew Rabanser, Stephan Allaire, J. J. Bommasani, Rishi Coppock, Harry Dubois, Magda Hadfield, Gillian K Hall, Andrew B. Hooker, Sara Lazar, Seth Newman, Steve Papailiopoulos, Dimitris Tekofsky, Shoshannah Toner, Helen Ududec, Cozmin Narayanan, Arvind |
| author_facet | Kapoor, Sayash Kirgis, Peter Schwartz, Andrew Rabanser, Stephan Allaire, J. J. Bommasani, Rishi Coppock, Harry Dubois, Magda Hadfield, Gillian K Hall, Andrew B. Hooker, Sara Lazar, Seth Newman, Steve Papailiopoulos, Dimitris Tekofsky, Shoshannah Toner, Helen Ududec, Cozmin Narayanan, Arvind |
| contents | Benchmark-based evaluation remains important for tracking frontier AI progress. But it can both overstate and understate deployed capability because it privileges tasks that can be precisely specified, automatically graded, easy to optimize for, and run with low budgets and short time horizons. We advocate for a complementary class of evaluations, which we term open-world evaluations: long-horizon, messy, real-world tasks assessed through small-sample qualitative analysis rather than benchmark-scale automation. In this paper we survey recent open-world evaluations, identify their strengths and limitations, and introduce CRUX (Collaborative Research for Updating AI eXpectations), a project for conducting such evaluations regularly. As a first instance, we task an AI agent with developing and publishing a simple iOS application to the Apple App Store. The agent completed the task with only a single avoidable manual intervention, suggesting that open-world evaluations can provide early warning of capabilities that may soon become widespread. We conclude with recommendations for designing and reporting open-world evals. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20520 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Open-World Evaluations for Measuring Frontier AI Capabilities Kapoor, Sayash Kirgis, Peter Schwartz, Andrew Rabanser, Stephan Allaire, J. J. Bommasani, Rishi Coppock, Harry Dubois, Magda Hadfield, Gillian K Hall, Andrew B. Hooker, Sara Lazar, Seth Newman, Steve Papailiopoulos, Dimitris Tekofsky, Shoshannah Toner, Helen Ududec, Cozmin Narayanan, Arvind Artificial Intelligence Benchmark-based evaluation remains important for tracking frontier AI progress. But it can both overstate and understate deployed capability because it privileges tasks that can be precisely specified, automatically graded, easy to optimize for, and run with low budgets and short time horizons. We advocate for a complementary class of evaluations, which we term open-world evaluations: long-horizon, messy, real-world tasks assessed through small-sample qualitative analysis rather than benchmark-scale automation. In this paper we survey recent open-world evaluations, identify their strengths and limitations, and introduce CRUX (Collaborative Research for Updating AI eXpectations), a project for conducting such evaluations regularly. As a first instance, we task an AI agent with developing and publishing a simple iOS application to the Apple App Store. The agent completed the task with only a single avoidable manual intervention, suggesting that open-world evaluations can provide early warning of capabilities that may soon become widespread. We conclude with recommendations for designing and reporting open-world evals. |
| title | Open-World Evaluations for Measuring Frontier AI Capabilities |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2605.20520 |