Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2412.14161 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908529194958848 |
|---|---|
| author | Xu, Frank F. Song, Yufan Li, Boxuan Tang, Yuxuan Jain, Kritanjali Bao, Mengxue Wang, Zora Z. Zhou, Xuhui Guo, Zhitong Cao, Murong Yang, Mingyang Lu, Hao Yang Martin, Amaad Su, Zhe Maben, Leander Mehta, Raj Chi, Wayne Jang, Lawrence Xie, Yiqing Zhou, Shuyan Neubig, Graham |
| author_facet | Xu, Frank F. Song, Yufan Li, Boxuan Tang, Yuxuan Jain, Kritanjali Bao, Mengxue Wang, Zora Z. Zhou, Xuhui Guo, Zhitong Cao, Murong Yang, Mingyang Lu, Hao Yang Martin, Amaad Su, Zhe Maben, Leander Mehta, Raj Chi, Wayne Jang, Lawrence Xie, Yiqing Zhou, Shuyan Neubig, Graham |
| contents | We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at accelerating or even autonomously performing work-related tasks? The answer to this question has important implications both for industry looking to adopt AI into their workflows and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that the most competitive agent can complete 30% of tasks autonomously. This paints a nuanced picture on task automation with LM agents--in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems. We release code, data, environment, and experiments on https://the-agent-company.com. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14161 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks Xu, Frank F. Song, Yufan Li, Boxuan Tang, Yuxuan Jain, Kritanjali Bao, Mengxue Wang, Zora Z. Zhou, Xuhui Guo, Zhitong Cao, Murong Yang, Mingyang Lu, Hao Yang Martin, Amaad Su, Zhe Maben, Leander Mehta, Raj Chi, Wayne Jang, Lawrence Xie, Yiqing Zhou, Shuyan Neubig, Graham Computation and Language We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at accelerating or even autonomously performing work-related tasks? The answer to this question has important implications both for industry looking to adopt AI into their workflows and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that the most competitive agent can complete 30% of tasks autonomously. This paints a nuanced picture on task automation with LM agents--in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems. We release code, data, environment, and experiments on https://the-agent-company.com. |
| title | TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.14161 |