OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916811913560064 |
|---|---|
| author | Sun, Qiushi Cheng, Kanzhi Ding, Zichen Jin, Chuanyang Wang, Yian Xu, Fangzhi Wu, Zhenyu Jia, Chengyou Chen, Liheng Liu, Zhoumianze Kao, Ben Li, Guohao He, Junxian Qiao, Yu Wu, Zhiyong |
| author_facet | Sun, Qiushi Cheng, Kanzhi Ding, Zichen Jin, Chuanyang Wang, Yian Xu, Fangzhi Wu, Zhenyu Jia, Chengyou Chen, Liheng Liu, Zhoumianze Kao, Ben Li, Guohao He, Junxian Qiao, Yu Wu, Zhiyong |
| contents | Graphical User Interface (GUI) agents powered by Vision-Language Models (VLMs) have demonstrated human-like computer control capability. Despite their utility in advancing digital automation, a critical bottleneck persists: collecting high-quality trajectory data for training. Common practices for collecting such data rely on human supervision or synthetic data generation through executing pre-defined tasks, which are either resource-intensive or unable to guarantee data quality. Moreover, these methods suffer from limited data diversity and significant gaps between synthetic data and real-world environments. To address these challenges, we propose OS-Genesis, a novel GUI data synthesis pipeline that reverses the conventional trajectory collection process. Instead of relying on pre-defined tasks, OS-Genesis enables agents first to perceive environments and perform step-wise interactions, then retrospectively derive high-quality tasks to enable trajectory-level exploration. A trajectory reward model is then employed to ensure the quality of the generated trajectories. We demonstrate that training GUI agents with OS-Genesis significantly improves their performance on highly challenging online benchmarks. In-depth analysis further validates OS-Genesis's efficiency and its superior data quality and diversity compared to existing synthesis methods. Our codes, data, and checkpoints are available at https://qiushisun.github.io/OS-Genesis-Home/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_19723 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis Sun, Qiushi Cheng, Kanzhi Ding, Zichen Jin, Chuanyang Wang, Yian Xu, Fangzhi Wu, Zhenyu Jia, Chengyou Chen, Liheng Liu, Zhoumianze Kao, Ben Li, Guohao He, Junxian Qiao, Yu Wu, Zhiyong Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction Graphical User Interface (GUI) agents powered by Vision-Language Models (VLMs) have demonstrated human-like computer control capability. Despite their utility in advancing digital automation, a critical bottleneck persists: collecting high-quality trajectory data for training. Common practices for collecting such data rely on human supervision or synthetic data generation through executing pre-defined tasks, which are either resource-intensive or unable to guarantee data quality. Moreover, these methods suffer from limited data diversity and significant gaps between synthetic data and real-world environments. To address these challenges, we propose OS-Genesis, a novel GUI data synthesis pipeline that reverses the conventional trajectory collection process. Instead of relying on pre-defined tasks, OS-Genesis enables agents first to perceive environments and perform step-wise interactions, then retrospectively derive high-quality tasks to enable trajectory-level exploration. A trajectory reward model is then employed to ensure the quality of the generated trajectories. We demonstrate that training GUI agents with OS-Genesis significantly improves their performance on highly challenging online benchmarks. In-depth analysis further validates OS-Genesis's efficiency and its superior data quality and diversity compared to existing synthesis methods. Our codes, data, and checkpoints are available at https://qiushisun.github.io/OS-Genesis-Home/. |
| title | OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2412.19723 |