Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908941931249664 |
|---|---|
| author | Zhang, Yuzhe Xue, Xianwei Wu, Xingyong Chen, Mengke Liu, Chen He, Xinran Shao, Run Liu, Feiran Xu, Huanmin Pan, Qiutong Wang, Haiwei |
| author_facet | Zhang, Yuzhe Xue, Xianwei Wu, Xingyong Chen, Mengke Liu, Chen He, Xinran Shao, Run Liu, Feiran Xu, Huanmin Pan, Qiutong Wang, Haiwei |
| contents | Autonomous GUI agents based on vision-language models (VLMs) often assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. In real-world settings with network latency, rendering delays, and system interruptions, this assumption leads to undetected action failures, repetitive ineffective behaviors, and catastrophic error accumulation. Moreover, learning robust recovery strategies is challenging due to the high cost of online interaction and the lack of real-time feedback in offline datasets.We propose VeriGUI (Verification-driven GUI Agent), which explicitly models action outcomes and recovery under noisy environments. VeriGUI introduces a Thinking--Verification--Action--Expectation (TVAE) framework to detect failures and guide corrective reasoning, and a two-stage training pipeline that combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards. We further construct a Robustness Benchmark based on AndroidControl to evaluate failure recognition and correction. Experiments show that VeriGUI significantly reduces failure loops and improves recovery success while maintaining competitive standard task performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_05477 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction Zhang, Yuzhe Xue, Xianwei Wu, Xingyong Chen, Mengke Liu, Chen He, Xinran Shao, Run Liu, Feiran Xu, Huanmin Pan, Qiutong Wang, Haiwei Computation and Language Autonomous GUI agents based on vision-language models (VLMs) often assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. In real-world settings with network latency, rendering delays, and system interruptions, this assumption leads to undetected action failures, repetitive ineffective behaviors, and catastrophic error accumulation. Moreover, learning robust recovery strategies is challenging due to the high cost of online interaction and the lack of real-time feedback in offline datasets.We propose VeriGUI (Verification-driven GUI Agent), which explicitly models action outcomes and recovery under noisy environments. VeriGUI introduces a Thinking--Verification--Action--Expectation (TVAE) framework to detect failures and guide corrective reasoning, and a two-stage training pipeline that combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards. We further construct a Robustness Benchmark based on AndroidControl to evaluate failure recognition and correction. Experiments show that VeriGUI significantly reduces failure loops and improves recovery success while maintaining competitive standard task performance. |
| title | Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.05477 |