Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916974766850048 |
|---|---|
| author | Li, Pengxiang Hu, Zechen Shang, Zirui Wu, Jingrong Liu, Yang Liu, Hui Gao, Zhi Shi, Chenrui Zhang, Bofei Zhang, Zihao Shi, Xiaochuan YU, Zedong Wu, Yuwei Wu, Xinxiao Jia, Yunde Xiang, Liuyu He, Zhaofeng Li, Qing |
| author_facet | Li, Pengxiang Hu, Zechen Shang, Zirui Wu, Jingrong Liu, Yang Liu, Hui Gao, Zhi Shi, Chenrui Zhang, Bofei Zhang, Zihao Shi, Xiaochuan YU, Zedong Wu, Yuwei Wu, Xinxiao Jia, Yunde Xiang, Liuyu He, Zhaofeng Li, Qing |
| contents | Vision-language model (VLM) based GUI agents show promise for automating complex desktop and mobile tasks, but face significant challenges in applying reinforcement learning (RL): (1) slow multi-turn interactions with GUI environments for policy rollout, and (2) insufficient high-quality agent-environment interactions for policy learning. To address these challenges, we propose DART, a Decoupled Agentic RL Training framework for GUI agents, which coordinates heterogeneous modules in a highly decoupled manner. DART separates the training system into four asynchronous modules: environment cluster, rollout service, data manager, and trainer. This design enables non-blocking communication, asynchronous training, rollout-wise trajectory sampling, and per-worker model synchronization, significantly improving the system efficiency: 1.6*GPU utilization for rollout, 1.9* training throughput, and 5.5* environment utilization. To facilitate effective learning from abundant samples, we introduce an adaptive data curation scheme: (1) pre-collecting successful trajectories for challenging tasks to supplement sparse success in online sampling; (2) dynamically adjusting rollout numbers and trajectory lengths based on task difficulty; (3) training selectively on high-entropy steps to prioritize critical decisions; (4) stabilizing learning via truncated importance sampling for policy mismatch between policy rollout and updating. On the OSWorld benchmark, DART-GUI-7B achieves a 42.13% task success rate, a 14.61% absolute gain over the base model, and 7.34% higher than open-source SOTA. We will fully open-source our training framework, data, and model checkpoints via computer-use-agents.github.io/dart-gui, which we believe is a timely contribution to the open-source community of agentic RL training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23866 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation Li, Pengxiang Hu, Zechen Shang, Zirui Wu, Jingrong Liu, Yang Liu, Hui Gao, Zhi Shi, Chenrui Zhang, Bofei Zhang, Zihao Shi, Xiaochuan YU, Zedong Wu, Yuwei Wu, Xinxiao Jia, Yunde Xiang, Liuyu He, Zhaofeng Li, Qing Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Vision-language model (VLM) based GUI agents show promise for automating complex desktop and mobile tasks, but face significant challenges in applying reinforcement learning (RL): (1) slow multi-turn interactions with GUI environments for policy rollout, and (2) insufficient high-quality agent-environment interactions for policy learning. To address these challenges, we propose DART, a Decoupled Agentic RL Training framework for GUI agents, which coordinates heterogeneous modules in a highly decoupled manner. DART separates the training system into four asynchronous modules: environment cluster, rollout service, data manager, and trainer. This design enables non-blocking communication, asynchronous training, rollout-wise trajectory sampling, and per-worker model synchronization, significantly improving the system efficiency: 1.6*GPU utilization for rollout, 1.9* training throughput, and 5.5* environment utilization. To facilitate effective learning from abundant samples, we introduce an adaptive data curation scheme: (1) pre-collecting successful trajectories for challenging tasks to supplement sparse success in online sampling; (2) dynamically adjusting rollout numbers and trajectory lengths based on task difficulty; (3) training selectively on high-entropy steps to prioritize critical decisions; (4) stabilizing learning via truncated importance sampling for policy mismatch between policy rollout and updating. On the OSWorld benchmark, DART-GUI-7B achieves a 42.13% task success rate, a 14.61% absolute gain over the base model, and 7.34% higher than open-source SOTA. We will fully open-source our training framework, data, and model checkpoints via computer-use-agents.github.io/dart-gui, which we believe is a timely contribution to the open-source community of agentic RL training. |
| title | Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.23866 |