Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.24180 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916912660742144 |
|---|---|
| author | Cheng, Ziming Huang, Zhiyuan Pan, Junting Hou, Zhaohui Zhan, Mingjie |
| author_facet | Cheng, Ziming Huang, Zhiyuan Pan, Junting Hou, Zhaohui Zhan, Mingjie |
| contents | Graphical user interfaces (GUI) automation agents are emerging as powerful tools, enabling humans to accomplish increasingly complex tasks on smart devices. However, users often inadvertently omit key information when conveying tasks, which hinders agent performance in the current agent paradigm that does not support immediate user intervention. To address this issue, we introduce a $\textbf{Self-Correction GUI Navigation}$ task that incorporates interactive information completion capabilities within GUI agents. We developed the $\textbf{Navi-plus}$ dataset with GUI follow-up question-answer pairs, alongside a $\textbf{Dual-Stream Trajectory Evaluation}$ method to benchmark this new capability. Our results show that agents equipped with the ability to ask GUI follow-up questions can fully recover their performance when faced with ambiguous user tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_24180 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questions Cheng, Ziming Huang, Zhiyuan Pan, Junting Hou, Zhaohui Zhan, Mingjie Computer Vision and Pattern Recognition Human-Computer Interaction Graphical user interfaces (GUI) automation agents are emerging as powerful tools, enabling humans to accomplish increasingly complex tasks on smart devices. However, users often inadvertently omit key information when conveying tasks, which hinders agent performance in the current agent paradigm that does not support immediate user intervention. To address this issue, we introduce a $\textbf{Self-Correction GUI Navigation}$ task that incorporates interactive information completion capabilities within GUI agents. We developed the $\textbf{Navi-plus}$ dataset with GUI follow-up question-answer pairs, alongside a $\textbf{Dual-Stream Trajectory Evaluation}$ method to benchmark this new capability. Our results show that agents equipped with the ability to ask GUI follow-up questions can fully recover their performance when faced with ambiguous user tasks. |
| title | Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questions |
| topic | Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2503.24180 |