WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929431234215936 |
|---|---|
| author | Drouin, Alexandre Gasse, Maxime Caccia, Massimo Laradji, Issam H. Del Verme, Manuel Marty, Tom Boisvert, Léo Thakkar, Megh Cappart, Quentin Vazquez, David Chapados, Nicolas Lacoste, Alexandre |
| author_facet | Drouin, Alexandre Gasse, Maxime Caccia, Massimo Laradji, Issam H. Del Verme, Manuel Marty, Tom Boisvert, Léo Thakkar, Megh Cappart, Quentin Vazquez, David Chapados, Nicolas Lacoste, Alexandre |
| contents | We study the use of large language model-based agents for interacting with software via web browsers. Unlike prior work, we focus on measuring the agents' ability to perform tasks that span the typical daily work of knowledge workers utilizing enterprise software systems. To this end, we propose WorkArena, a remote-hosted benchmark of 33 tasks based on the widely-used ServiceNow platform. We also introduce BrowserGym, an environment for the design and evaluation of such agents, offering a rich set of actions as well as multimodal observations. Our empirical evaluation reveals that while current agents show promise on WorkArena, there remains a considerable gap towards achieving full task automation. Notably, our analysis uncovers a significant performance disparity between open and closed-source LLMs, highlighting a critical area for future exploration and development in the field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07718 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? Drouin, Alexandre Gasse, Maxime Caccia, Massimo Laradji, Issam H. Del Verme, Manuel Marty, Tom Boisvert, Léo Thakkar, Megh Cappart, Quentin Vazquez, David Chapados, Nicolas Lacoste, Alexandre Machine Learning Artificial Intelligence We study the use of large language model-based agents for interacting with software via web browsers. Unlike prior work, we focus on measuring the agents' ability to perform tasks that span the typical daily work of knowledge workers utilizing enterprise software systems. To this end, we propose WorkArena, a remote-hosted benchmark of 33 tasks based on the widely-used ServiceNow platform. We also introduce BrowserGym, an environment for the design and evaluation of such agents, offering a rich set of actions as well as multimodal observations. Our empirical evaluation reveals that while current agents show promise on WorkArena, there remains a considerable gap towards achieving full task automation. Notably, our analysis uncovers a significant performance disparity between open and closed-source LLMs, highlighting a critical area for future exploration and development in the field. |
| title | WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.07718 |