Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2510.27266 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910262255157248 |
|---|---|
| author | Zhang, Shaojie Fu, Pei Zhang, Ruoceng Yang, Jiahui Du, Anan Xi, Xiuwen Wang, Shaokang Huang, Ying Qin, Bin Luo, Zhenbo Luan, Jian |
| author_facet | Zhang, Shaojie Fu, Pei Zhang, Ruoceng Yang, Jiahui Du, Anan Xi, Xiuwen Wang, Shaokang Huang, Ying Qin, Bin Luo, Zhenbo Luan, Jian |
| contents | Autonomous graphical user interface (GUI) agents rely on accurate GUI grounding, which maps language instructions to on-screen coordinates, to execute user commands. However, current models, whether trained via supervised fine-tuning (SFT) or reinforcement learning (RL), often provide confidence signals that are poorly aligned with actual grounding correctness, leading to overconfident and unreliable predictions. To address this, we propose HyperClick, a novel framework that enhances trustworthy GUI grounding through self-critiqued reinforcement learning (SCRL). HyperClick combines a correctness reward and a confidence alignment reward, training the policy model to output both a click prediction and an explicit confidence estimate. This approach jointly optimizes grounding accuracy and confidence reliability through confidence-based self-assessment. Extensive experiments on challenging benchmarks show that HyperClick maintains strong grounding performance while providing better-aligned confidence estimates. By exposing uncertainty alongside GUI actions, HyperClick supports confidence-based abstention in GUI automation. Code will be released here. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_27266 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning Zhang, Shaojie Fu, Pei Zhang, Ruoceng Yang, Jiahui Du, Anan Xi, Xiuwen Wang, Shaokang Huang, Ying Qin, Bin Luo, Zhenbo Luan, Jian Computer Vision and Pattern Recognition Autonomous graphical user interface (GUI) agents rely on accurate GUI grounding, which maps language instructions to on-screen coordinates, to execute user commands. However, current models, whether trained via supervised fine-tuning (SFT) or reinforcement learning (RL), often provide confidence signals that are poorly aligned with actual grounding correctness, leading to overconfident and unreliable predictions. To address this, we propose HyperClick, a novel framework that enhances trustworthy GUI grounding through self-critiqued reinforcement learning (SCRL). HyperClick combines a correctness reward and a confidence alignment reward, training the policy model to output both a click prediction and an explicit confidence estimate. This approach jointly optimizes grounding accuracy and confidence reliability through confidence-based self-assessment. Extensive experiments on challenging benchmarks show that HyperClick maintains strong grounding performance while providing better-aligned confidence estimates. By exposing uncertainty alongside GUI actions, HyperClick supports confidence-based abstention in GUI automation. Code will be released here. |
| title | Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.27266 |