Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.17354 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915208372420608 |
|---|---|
| author | Rein, David Becker, Joel Deng, Amy Nix, Seraphina Canal, Chris O'Connel, Daniel Arnott, Pip Bloom, Ryan Broadley, Thomas Garcia, Katharyn Goodrich, Brian Hasin, Max Jawhar, Sami Kinniment, Megan Kwa, Thomas Lajko, Aron Rush, Nate Sato, Lucas Jun Koba Von Arx, Sydney West, Ben Chan, Lawrence Barnes, Elizabeth |
| author_facet | Rein, David Becker, Joel Deng, Amy Nix, Seraphina Canal, Chris O'Connel, Daniel Arnott, Pip Bloom, Ryan Broadley, Thomas Garcia, Katharyn Goodrich, Brian Hasin, Max Jawhar, Sami Kinniment, Megan Kwa, Thomas Lajko, Aron Rush, Nate Sato, Lucas Jun Koba Von Arx, Sydney West, Ben Chan, Lawrence Barnes, Elizabeth |
| contents | To understand and predict the societal impacts of highly autonomous AI systems, we need benchmarks with grounding, i.e., metrics that directly connect AI performance to real-world effects we care about. We present HCAST (Human-Calibrated Autonomy Software Tasks), a benchmark of 189 machine learning engineering, cybersecurity, software engineering, and general reasoning tasks. We collect 563 human baselines (totaling over 1500 hours) from people skilled in these domains, working under identical conditions as AI agents, which lets us estimate that HCAST tasks take humans between one minute and 8+ hours. Measuring the time tasks take for humans provides an intuitive metric for evaluating AI capabilities, helping answer the question "can an agent be trusted to complete a task that would take a human X hours?" We evaluate the success rates of AI agents built on frontier foundation models, and we find that current agents succeed 70-80% of the time on tasks that take humans less than one hour, and less than 20% of the time on tasks that take humans more than 4 hours. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_17354 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HCAST: Human-Calibrated Autonomy Software Tasks Rein, David Becker, Joel Deng, Amy Nix, Seraphina Canal, Chris O'Connel, Daniel Arnott, Pip Bloom, Ryan Broadley, Thomas Garcia, Katharyn Goodrich, Brian Hasin, Max Jawhar, Sami Kinniment, Megan Kwa, Thomas Lajko, Aron Rush, Nate Sato, Lucas Jun Koba Von Arx, Sydney West, Ben Chan, Lawrence Barnes, Elizabeth Artificial Intelligence I.2.0 To understand and predict the societal impacts of highly autonomous AI systems, we need benchmarks with grounding, i.e., metrics that directly connect AI performance to real-world effects we care about. We present HCAST (Human-Calibrated Autonomy Software Tasks), a benchmark of 189 machine learning engineering, cybersecurity, software engineering, and general reasoning tasks. We collect 563 human baselines (totaling over 1500 hours) from people skilled in these domains, working under identical conditions as AI agents, which lets us estimate that HCAST tasks take humans between one minute and 8+ hours. Measuring the time tasks take for humans provides an intuitive metric for evaluating AI capabilities, helping answer the question "can an agent be trusted to complete a task that would take a human X hours?" We evaluate the success rates of AI agents built on frontier foundation models, and we find that current agents succeed 70-80% of the time on tasks that take humans less than one hour, and less than 20% of the time on tasks that take humans more than 4 hours. |
| title | HCAST: Human-Calibrated Autonomy Software Tasks |
| topic | Artificial Intelligence I.2.0 |
| url | https://arxiv.org/abs/2503.17354 |