CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915820689424384 |
|---|---|
| author | Huq, Faria Wang, Zora Zhiruo Xu, Frank F. Ou, Tianyue Zhou, Shuyan Bigham, Jeffrey P. Neubig, Graham |
| author_facet | Huq, Faria Wang, Zora Zhiruo Xu, Frank F. Ou, Tianyue Zhou, Shuyan Bigham, Jeffrey P. Neubig, Graham |
| contents | While much work on web agents emphasizes the promise of autonomously performing tasks on behalf of users, in reality, agents often fall short on complex tasks in real-world contexts and modeling user preference. This presents an opportunity for humans to collaborate with the agent and leverage the agent's capabilities effectively. We propose CowPilot, a framework supporting autonomous as well as human-agent collaborative web navigation, and evaluation across task success and task efficiency. CowPilot reduces the number of steps humans need to perform by allowing agents to propose next steps, while users are able to pause, reject, or take alternative actions. During execution, users can interleave their actions with the agent by overriding suggestions or resuming agent control when needed. We conducted case studies on five common websites and found that the human-agent collaborative mode achieves the highest success rate of 95% while requiring humans to perform only 15.2% of the total steps. Even with human interventions during task execution, the agent successfully drives up to half of task success on its own. CowPilot can serve as a useful tool for data collection and agent evaluation across websites, which we believe will enable research in how users and agents can work together. Video demonstrations are available at https://oaishi.github.io/cowpilot.html |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16609 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation Huq, Faria Wang, Zora Zhiruo Xu, Frank F. Ou, Tianyue Zhou, Shuyan Bigham, Jeffrey P. Neubig, Graham Artificial Intelligence Computation and Language Human-Computer Interaction While much work on web agents emphasizes the promise of autonomously performing tasks on behalf of users, in reality, agents often fall short on complex tasks in real-world contexts and modeling user preference. This presents an opportunity for humans to collaborate with the agent and leverage the agent's capabilities effectively. We propose CowPilot, a framework supporting autonomous as well as human-agent collaborative web navigation, and evaluation across task success and task efficiency. CowPilot reduces the number of steps humans need to perform by allowing agents to propose next steps, while users are able to pause, reject, or take alternative actions. During execution, users can interleave their actions with the agent by overriding suggestions or resuming agent control when needed. We conducted case studies on five common websites and found that the human-agent collaborative mode achieves the highest success rate of 95% while requiring humans to perform only 15.2% of the total steps. Even with human interventions during task execution, the agent successfully drives up to half of task success on its own. CowPilot can serve as a useful tool for data collection and agent evaluation across websites, which we believe will enable research in how users and agents can work together. Video demonstrations are available at https://oaishi.github.io/cowpilot.html |
| title | CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation |
| topic | Artificial Intelligence Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2501.16609 |