How Well Does Agent Development Reflect Real-World Work?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911492175036416 |
|---|---|
| author | Wang, Zora Zhiruo Vijayvargiya, Sanidhya Chen, Aspen Zhang, Hanmo Arangarajan, Venu Arvind Chen, Jett Chen, Valerie Yang, Diyi Fried, Daniel Neubig, Graham |
| author_facet | Wang, Zora Zhiruo Vijayvargiya, Sanidhya Chen, Aspen Zhang, Hanmo Arangarajan, Venu Arvind Chen, Jett Chen, Valerie Yang, Diyi Fried, Daniel Neubig, Graham |
| contents | AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_01203 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Well Does Agent Development Reflect Real-World Work? Wang, Zora Zhiruo Vijayvargiya, Sanidhya Chen, Aspen Zhang, Hanmo Arangarajan, Venu Arvind Chen, Jett Chen, Valerie Yang, Diyi Fried, Daniel Neubig, Graham Artificial Intelligence AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation. |
| title | How Well Does Agent Development Reflect Real-World Work? |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.01203 |