Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2507.05791 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915532378210304 |
|---|---|
| author | Yang, Yan Li, Dongxu Dai, Yutong Yang, Yuhao Luo, Ziyang Zhao, Zirui Hu, Zhiyuan Huang, Junzhe Saha, Amrita Chen, Zeyuan Xu, Ran Pan, Liyuan Savarese, Silvio Xiong, Caiming Li, Junnan |
| author_facet | Yang, Yan Li, Dongxu Dai, Yutong Yang, Yuhao Luo, Ziyang Zhao, Zirui Hu, Zhiyuan Huang, Junzhe Saha, Amrita Chen, Zeyuan Xu, Ran Pan, Liyuan Savarese, Silvio Xiong, Caiming Li, Junnan |
| contents | Graphical user interface (GUI) agents autonomously complete tasks across platforms (\eg, Linux) by sequentially decomposing user instructions into action proposals that iteratively interact with visual elements in the evolving environment. However, two main challenges arise: i) planning (\ie, the action proposal sequence) under expansive action space, where selecting an appropriate plan is non-trivial, as many valid ones may exist; ii) accurately grounding actions in complex and high-resolution interfaces, \ie, precisely interacting with visual targets. This paper investigates the aforementioned challenges with our \textbf{G}UI \textbf{T}est-time Scaling \textbf{A}gent, namely GTA1. First, we conduct test-time scaling to select the most appropriate action proposal: at each step, multiple candidate proposals are sampled and evaluated and selected by a judge model. It trades off computation for better decision quality by concurrent sampling. Second, we propose a model that improves grounding of the selected action proposals to its corresponding visual elements. Our key insight is that reinforcement learning (RL) facilitates grounding through inherent objective alignments, rewarding successful clicks on interface elements. Experimentally, GTA1 achieves state-of-the-art performance on both grounding and agent task execution benchmarks. The code and models are released here. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_05791 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GTA1: GUI Test-time Scaling Agent Yang, Yan Li, Dongxu Dai, Yutong Yang, Yuhao Luo, Ziyang Zhao, Zirui Hu, Zhiyuan Huang, Junzhe Saha, Amrita Chen, Zeyuan Xu, Ran Pan, Liyuan Savarese, Silvio Xiong, Caiming Li, Junnan Artificial Intelligence Graphical user interface (GUI) agents autonomously complete tasks across platforms (\eg, Linux) by sequentially decomposing user instructions into action proposals that iteratively interact with visual elements in the evolving environment. However, two main challenges arise: i) planning (\ie, the action proposal sequence) under expansive action space, where selecting an appropriate plan is non-trivial, as many valid ones may exist; ii) accurately grounding actions in complex and high-resolution interfaces, \ie, precisely interacting with visual targets. This paper investigates the aforementioned challenges with our \textbf{G}UI \textbf{T}est-time Scaling \textbf{A}gent, namely GTA1. First, we conduct test-time scaling to select the most appropriate action proposal: at each step, multiple candidate proposals are sampled and evaluated and selected by a judge model. It trades off computation for better decision quality by concurrent sampling. Second, we propose a model that improves grounding of the selected action proposals to its corresponding visual elements. Our key insight is that reinforcement learning (RL) facilitates grounding through inherent objective alignments, rewarding successful clicks on interface elements. Experimentally, GTA1 achieves state-of-the-art performance on both grounding and agent task execution benchmarks. The code and models are released here. |
| title | GTA1: GUI Test-time Scaling Agent |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2507.05791 |