ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929616583655424 |
|---|---|
| author | Shen, Junhong Jain, Atishay Xiao, Zedian Amlekar, Ishan Hadji, Mouad Podolny, Aaron Talwalkar, Ameet |
| author_facet | Shen, Junhong Jain, Atishay Xiao, Zedian Amlekar, Ishan Hadji, Mouad Podolny, Aaron Talwalkar, Ameet |
| contents | Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks -- ScribeAgent achieves state-of-the-art direct generation performance on Mind2Web and improves the task success rate by 7.3% over the previous best text-only web agents on WebArena. We further perform detailed ablation studies on various fine-tuning design choices and provide insights into LLM selection, training recipes, context window optimization, and effect of dataset sizes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_15004 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data Shen, Junhong Jain, Atishay Xiao, Zedian Amlekar, Ishan Hadji, Mouad Podolny, Aaron Talwalkar, Ameet Computation and Language Artificial Intelligence Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks -- ScribeAgent achieves state-of-the-art direct generation performance on Mind2Web and improves the task success rate by 7.3% over the previous best text-only web agents on WebArena. We further perform detailed ablation studies on various fine-tuning design choices and provide insights into LLM selection, training recipes, context window optimization, and effect of dataset sizes. |
| title | ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2411.15004 |