TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913860898783232 |
|---|---|
| author | Lu, Yuheng Yu, Qian Wang, Hongru Liu, Zeming Su, Wei Liu, Yanping Guo, Yuhang Liang, Maocheng Wang, Yunhong Wang, Haifeng |
| author_facet | Lu, Yuheng Yu, Qian Wang, Hongru Liu, Zeming Su, Wei Liu, Yanping Guo, Yuhang Liang, Maocheng Wang, Yunhong Wang, Haifeng |
| contents | Graphical User Interface (GUI) agents, which autonomously operate on digital interfaces through natural language instructions, hold transformative potential for accessibility, automation, and user experience. A critical aspect of their functionality is grounding - the ability to map linguistic intents to visual and structural interface elements. However, existing GUI agents often struggle to adapt to the dynamic and interconnected nature of real-world digital environments, where tasks frequently span multiple platforms and applications while also being impacted by version updates. To address this, we introduce TransBench, the first benchmark designed to systematically evaluate and enhance the transferability of GUI agents across three key dimensions: cross-version transferability (adapting to version updates), cross-platform transferability (generalizing across platforms like iOS, Android, and Web), and cross-application transferability (handling tasks spanning functionally distinct apps). TransBench includes 15 app categories with diverse functionalities, capturing essential pages across versions and platforms to enable robust evaluation. Our experiments demonstrate significant improvements in grounding accuracy, showcasing the practical utility of GUI agents in dynamic, real-world environments. Our code and data will be publicly available at GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17629 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments Lu, Yuheng Yu, Qian Wang, Hongru Liu, Zeming Su, Wei Liu, Yanping Guo, Yuhang Liang, Maocheng Wang, Yunhong Wang, Haifeng Human-Computer Interaction Artificial Intelligence Graphical User Interface (GUI) agents, which autonomously operate on digital interfaces through natural language instructions, hold transformative potential for accessibility, automation, and user experience. A critical aspect of their functionality is grounding - the ability to map linguistic intents to visual and structural interface elements. However, existing GUI agents often struggle to adapt to the dynamic and interconnected nature of real-world digital environments, where tasks frequently span multiple platforms and applications while also being impacted by version updates. To address this, we introduce TransBench, the first benchmark designed to systematically evaluate and enhance the transferability of GUI agents across three key dimensions: cross-version transferability (adapting to version updates), cross-platform transferability (generalizing across platforms like iOS, Android, and Web), and cross-application transferability (handling tasks spanning functionally distinct apps). TransBench includes 15 app categories with diverse functionalities, capturing essential pages across versions and platforms to enable robust evaluation. Our experiments demonstrate significant improvements in grounding accuracy, showcasing the practical utility of GUI agents in dynamic, real-world environments. Our code and data will be publicly available at GitHub. |
| title | TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2505.17629 |