VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918165591621632 |
|---|---|
| author | Lu, Dunjie Xu, Yiheng Wang, Junli Wu, Haoyuan Wang, Xinyuan Wang, Zekun Yang, Junlin Su, Hongjin Chen, Jixuan Chen, Junda Mao, Yuchen Zhou, Jingren Lin, Junyang Hui, Binyuan Yu, Tao |
| author_facet | Lu, Dunjie Xu, Yiheng Wang, Junli Wu, Haoyuan Wang, Xinyuan Wang, Zekun Yang, Junlin Su, Hongjin Chen, Jixuan Chen, Junda Mao, Yuchen Zhou, Jingren Lin, Junyang Hui, Binyuan Yu, Tao |
| contents | Training computer-use agents requires massive amounts of GUI interaction data, but manually annotating action trajectories at scale is prohibitively expensive. We present VideoAgentTrek, a scalable pipeline that automatically mines training data from publicly available screen-recorded videos at web scale, eliminating the need for manual annotation. Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels. To solve this, we develop Video2Action, an inverse dynamics module (IDM) with two components: (1) a video grounding model that detects and localizes GUI actions with precise temporal boundaries and context, and (2) an action-content recognizer that extracts structured parameters like click coordinates and typed text with high fidelity. Applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps automatically. We leverage this data through continued pretraining followed by supervised fine-tuning. On OSWorld-Verified, our approach improves task success rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On AgentNetBench, step accuracy increases from 64.1% to 69.3%. Our results demonstrate that passive internet videos can be transformed into high-quality supervision for computer-use agents, providing a scalable alternative to expensive manual annotation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_19488 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos Lu, Dunjie Xu, Yiheng Wang, Junli Wu, Haoyuan Wang, Xinyuan Wang, Zekun Yang, Junlin Su, Hongjin Chen, Jixuan Chen, Junda Mao, Yuchen Zhou, Jingren Lin, Junyang Hui, Binyuan Yu, Tao Computation and Language Artificial Intelligence Machine Learning Training computer-use agents requires massive amounts of GUI interaction data, but manually annotating action trajectories at scale is prohibitively expensive. We present VideoAgentTrek, a scalable pipeline that automatically mines training data from publicly available screen-recorded videos at web scale, eliminating the need for manual annotation. Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels. To solve this, we develop Video2Action, an inverse dynamics module (IDM) with two components: (1) a video grounding model that detects and localizes GUI actions with precise temporal boundaries and context, and (2) an action-content recognizer that extracts structured parameters like click coordinates and typed text with high fidelity. Applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps automatically. We leverage this data through continued pretraining followed by supervised fine-tuning. On OSWorld-Verified, our approach improves task success rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On AgentNetBench, step accuracy increases from 64.1% to 69.3%. Our results demonstrate that passive internet videos can be transformed into high-quality supervision for computer-use agents, providing a scalable alternative to expensive manual annotation. |
| title | VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2510.19488 |