Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.01676 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915914873569280 |
|---|---|
| author | Zhao, Zirui Liew, Jun Hao Yang, Yan Yang, Wenzhuo Luo, Ziyang Sahoo, Doyen Savarese, Silvio Li, Junnan |
| author_facet | Zhao, Zirui Liew, Jun Hao Yang, Yan Yang, Wenzhuo Luo, Ziyang Sahoo, Doyen Savarese, Silvio Li, Junnan |
| contents | GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3 Pro (with CUA tools) and found that GPA achieves higher success rate with 10 times faster execution speed in finishing long-horizon GUI tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_01676 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GPA: Learning GUI Process Automation from Demonstrations Zhao, Zirui Liew, Jun Hao Yang, Yan Yang, Wenzhuo Luo, Ziyang Sahoo, Doyen Savarese, Silvio Li, Junnan Computer Vision and Pattern Recognition Artificial Intelligence Software Engineering GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3 Pro (with CUA tools) and found that GPA achieves higher success rate with 10 times faster execution speed in finishing long-horizon GUI tasks. |
| title | GPA: Learning GUI Process Automation from Demonstrations |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2604.01676 |