WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913680342384640 |
|---|---|
| author | Boisvert, Léo Thakkar, Megh Gasse, Maxime Caccia, Massimo De Chezelles, Thibault Le Sellier Cappart, Quentin Chapados, Nicolas Lacoste, Alexandre Drouin, Alexandre |
| author_facet | Boisvert, Léo Thakkar, Megh Gasse, Maxime Caccia, Massimo De Chezelles, Thibault Le Sellier Cappart, Quentin Chapados, Nicolas Lacoste, Alexandre Drouin, Alexandre |
| contents | The ability of large language models (LLMs) to mimic human-like intelligence has led to a surge in LLM-based autonomous agents. Though recent LLMs seem capable of planning and reasoning given user instructions, their effectiveness in applying these capabilities for autonomous task solving remains underexplored. This is especially true in enterprise settings, where automated agents hold the promise of a high impact. To fill this gap, we propose WorkArena++, a novel benchmark consisting of 682 tasks corresponding to realistic workflows routinely performed by knowledge workers. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents. Our empirical studies across state-of-the-art LLMs and vision-language models (VLMs), as well as human workers, reveal several challenges for such models to serve as useful assistants in the workplace. In addition to the benchmark, we provide a mechanism to effortlessly generate thousands of ground-truth observation/action traces, which can be used for fine-tuning existing models. Overall, we expect this work to serve as a useful resource to help the community progress toward capable autonomous agents. The benchmark can be found at https://github.com/ServiceNow/WorkArena. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_05291 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks Boisvert, Léo Thakkar, Megh Gasse, Maxime Caccia, Massimo De Chezelles, Thibault Le Sellier Cappart, Quentin Chapados, Nicolas Lacoste, Alexandre Drouin, Alexandre Artificial Intelligence The ability of large language models (LLMs) to mimic human-like intelligence has led to a surge in LLM-based autonomous agents. Though recent LLMs seem capable of planning and reasoning given user instructions, their effectiveness in applying these capabilities for autonomous task solving remains underexplored. This is especially true in enterprise settings, where automated agents hold the promise of a high impact. To fill this gap, we propose WorkArena++, a novel benchmark consisting of 682 tasks corresponding to realistic workflows routinely performed by knowledge workers. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents. Our empirical studies across state-of-the-art LLMs and vision-language models (VLMs), as well as human workers, reveal several challenges for such models to serve as useful assistants in the workplace. In addition to the benchmark, we provide a mechanism to effortlessly generate thousands of ground-truth observation/action traces, which can be used for fine-tuning existing models. Overall, we expect this work to serve as a useful resource to help the community progress toward capable autonomous agents. The benchmark can be found at https://github.com/ServiceNow/WorkArena. |
| title | WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2407.05291 |