Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.12373 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909256409677824 |
|---|---|
| author | Pan, Yichen Kong, Dehan Zhou, Sida Cui, Cheng Leng, Yifei Jiang, Bing Liu, Hangyu Shang, Yanyi Zhou, Shuyan Wu, Tongshuang Wu, Zhengyang |
| author_facet | Pan, Yichen Kong, Dehan Zhou, Sida Cui, Cheng Leng, Yifei Jiang, Bing Liu, Hangyu Shang, Yanyi Zhou, Shuyan Wu, Tongshuang Wu, Zhengyang |
| contents | For web agents to be practically useful, they must adapt to the continuously evolving web environment characterized by frequent updates to user interfaces and content. However, most existing benchmarks only capture the static aspects of the web. To bridge this gap, we introduce WebCanvas, an innovative online evaluation framework for web agents that effectively addresses the dynamic nature of web interactions. WebCanvas contains three main components to facilitate realistic assessments: (1) A novel evaluation metric which reliably capture critical intermediate actions or states necessary for task completions while disregarding noise caused by insignificant events or changed web-elements. (2) A benchmark dataset called Mind2Web-Live, a refined version of original Mind2Web static dataset containing 542 tasks with 2439 intermediate evaluation states; (3) Lightweight and generalizable annotation tools and testing pipelines that enables the community to collect and maintain the high-quality, up-to-date dataset. Building on WebCanvas, we open-source an agent framework with extensible modules for reasoning, providing a foundation for the community to conduct online inference and evaluations. Our best-performing agent achieves a task success rate of 23.1% and a task completion rate of 48.8% on the Mind2Web-Live test set. Additionally, we analyze the performance discrepancies across various websites, domains, and experimental environments. We encourage the community to contribute further insights on online agent evaluation, thereby advancing this field of research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12373 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WebCanvas: Benchmarking Web Agents in Online Environments Pan, Yichen Kong, Dehan Zhou, Sida Cui, Cheng Leng, Yifei Jiang, Bing Liu, Hangyu Shang, Yanyi Zhou, Shuyan Wu, Tongshuang Wu, Zhengyang Computation and Language Artificial Intelligence Machine Learning 68T50 I.2.7 For web agents to be practically useful, they must adapt to the continuously evolving web environment characterized by frequent updates to user interfaces and content. However, most existing benchmarks only capture the static aspects of the web. To bridge this gap, we introduce WebCanvas, an innovative online evaluation framework for web agents that effectively addresses the dynamic nature of web interactions. WebCanvas contains three main components to facilitate realistic assessments: (1) A novel evaluation metric which reliably capture critical intermediate actions or states necessary for task completions while disregarding noise caused by insignificant events or changed web-elements. (2) A benchmark dataset called Mind2Web-Live, a refined version of original Mind2Web static dataset containing 542 tasks with 2439 intermediate evaluation states; (3) Lightweight and generalizable annotation tools and testing pipelines that enables the community to collect and maintain the high-quality, up-to-date dataset. Building on WebCanvas, we open-source an agent framework with extensible modules for reasoning, providing a foundation for the community to conduct online inference and evaluations. Our best-performing agent achieves a task success rate of 23.1% and a task completion rate of 48.8% on the Mind2Web-Live test set. Additionally, we analyze the performance discrepancies across various websites, domains, and experimental environments. We encourage the community to contribute further insights on online agent evaluation, thereby advancing this field of research. |
| title | WebCanvas: Benchmarking Web Agents in Online Environments |
| topic | Computation and Language Artificial Intelligence Machine Learning 68T50 I.2.7 |
| url | https://arxiv.org/abs/2406.12373 |