APEX-SWE
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910064551395328 |
|---|---|
| author | Kottamasu, Abhi Mahapatra, Chirag Lee, Sam Pan, Ben Barthwal, Aakash Datta, Akul Gupta, Anurag Mehta, Pranav Arun, Ajay Alberti, Silas Hiremath, Adarsh Foody, Brendan Vidgen, Bertie |
| author_facet | Kottamasu, Abhi Mahapatra, Chirag Lee, Sam Pan, Ben Barthwal, Aakash Datta, Akul Gupta, Anurag Mehta, Pranav Arun, Ajay Alberti, Silas Hiremath, Adarsh Foody, Brendan Vidgen, Bertie |
| contents | We introduce the AI Productivity Index for Software Engineering (APEX-SWE), a benchmark for assessing whether frontier AI models can execute economically valuable software engineering work. Unlike existing evaluations that focus on narrow, well-defined tasks, APEX-SWE assesses two novel task types that reflect real-world software engineering: (1) Integration tasks (n=100), which require constructing end-to-end systems across heterogeneous cloud primitives, business applications, and infrastructure-as-code services, and (2) Observability tasks (n=100), which require debugging production failures using telemetry signals such as logs and dashboards, as well as unstructured context. We evaluated eleven frontier models for the APEX-SWE leaderboard. Claude Opus 4.6 leads the APEX-SWE leaderboard with 40.5% Pass@1, followed by Claude Opus 4.5 at 38.7%. Our analysis shows that strong performance is primarily driven by epistemic discipline, defined as the capacity to distinguish between assumptions and verified facts. It is often combined with systematic verification prior to acting. We open-source the APEX-SWE evaluation harness and a dev set (n=50). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_08806 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | APEX-SWE Kottamasu, Abhi Mahapatra, Chirag Lee, Sam Pan, Ben Barthwal, Aakash Datta, Akul Gupta, Anurag Mehta, Pranav Arun, Ajay Alberti, Silas Hiremath, Adarsh Foody, Brendan Vidgen, Bertie Software Engineering Artificial Intelligence Computation and Language We introduce the AI Productivity Index for Software Engineering (APEX-SWE), a benchmark for assessing whether frontier AI models can execute economically valuable software engineering work. Unlike existing evaluations that focus on narrow, well-defined tasks, APEX-SWE assesses two novel task types that reflect real-world software engineering: (1) Integration tasks (n=100), which require constructing end-to-end systems across heterogeneous cloud primitives, business applications, and infrastructure-as-code services, and (2) Observability tasks (n=100), which require debugging production failures using telemetry signals such as logs and dashboards, as well as unstructured context. We evaluated eleven frontier models for the APEX-SWE leaderboard. Claude Opus 4.6 leads the APEX-SWE leaderboard with 40.5% Pass@1, followed by Claude Opus 4.5 at 38.7%. Our analysis shows that strong performance is primarily driven by epistemic discipline, defined as the capacity to distinguish between assumptions and verified facts. It is often combined with systematic verification prior to acting. We open-source the APEX-SWE evaluation harness and a dev set (n=50). |
| title | APEX-SWE |
| topic | Software Engineering Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2601.08806 |