GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915533890256896 |
|---|---|
| author | Patwardhan, Tejal Dias, Rachel Proehl, Elizabeth Kim, Grace Wang, Michele Watkins, Olivia Fishman, Simón Posada Aljubeh, Marwan Thacker, Phoebe Fauconnet, Laurance Kim, Natalie S. Chao, Patrick Miserendino, Samuel Chabot, Gildas Li, David Sharman, Michael Barr, Alexandra Glaese, Amelia Tworek, Jerry |
| author_facet | Patwardhan, Tejal Dias, Rachel Proehl, Elizabeth Kim, Grace Wang, Michele Watkins, Olivia Fishman, Simón Posada Aljubeh, Marwan Thacker, Phoebe Fauconnet, Laurance Kim, Natalie S. Chao, Patrick Miserendino, Samuel Chabot, Gildas Li, David Sharman, Michael Barr, Alexandra Glaese, Amelia Tworek, Jerry |
| contents | We introduce GDPval, a benchmark evaluating AI model capabilities on real-world economically valuable tasks. GDPval covers the majority of U.S. Bureau of Labor Statistics Work Activities for 44 occupations across the top 9 sectors contributing to U.S. GDP (Gross Domestic Product). Tasks are constructed from the representative work of industry professionals with an average of 14 years of experience. We find that frontier model performance on GDPval is improving roughly linearly over time, and that the current best frontier models are approaching industry experts in deliverable quality. We analyze the potential for frontier models, when paired with human oversight, to perform GDPval tasks cheaper and faster than unaided experts. We also demonstrate that increased reasoning effort, increased task context, and increased scaffolding improves model performance on GDPval. Finally, we open-source a gold subset of 220 tasks and provide a public automated grading service at evals.openai.com to facilitate future research in understanding real-world model capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04374 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks Patwardhan, Tejal Dias, Rachel Proehl, Elizabeth Kim, Grace Wang, Michele Watkins, Olivia Fishman, Simón Posada Aljubeh, Marwan Thacker, Phoebe Fauconnet, Laurance Kim, Natalie S. Chao, Patrick Miserendino, Samuel Chabot, Gildas Li, David Sharman, Michael Barr, Alexandra Glaese, Amelia Tworek, Jerry Machine Learning Artificial Intelligence Computers and Society We introduce GDPval, a benchmark evaluating AI model capabilities on real-world economically valuable tasks. GDPval covers the majority of U.S. Bureau of Labor Statistics Work Activities for 44 occupations across the top 9 sectors contributing to U.S. GDP (Gross Domestic Product). Tasks are constructed from the representative work of industry professionals with an average of 14 years of experience. We find that frontier model performance on GDPval is improving roughly linearly over time, and that the current best frontier models are approaching industry experts in deliverable quality. We analyze the potential for frontier models, when paired with human oversight, to perform GDPval tasks cheaper and faster than unaided experts. We also demonstrate that increased reasoning effort, increased task context, and increased scaffolding improves model performance on GDPval. Finally, we open-source a gold subset of 220 tasks and provide a public automated grading service at evals.openai.com to facilitate future research in understanding real-world model capabilities. |
| title | GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks |
| topic | Machine Learning Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2510.04374 |