Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913876453359616 |
|---|---|
| author | Chen, Chaoran Zhang, Zhiping Khalilov, Ibrahim Guo, Bingcan Gebreegziabher, Simret A Ye, Yanfang Xiao, Ziang Yao, Yaxing Li, Tianshi Li, Toby Jia-Jun |
| author_facet | Chen, Chaoran Zhang, Zhiping Khalilov, Ibrahim Guo, Bingcan Gebreegziabher, Simret A Ye, Yanfang Xiao, Ziang Yao, Yaxing Li, Tianshi Li, Toby Jia-Jun |
| contents | The rise of Large Language Models (LLMs) has revolutionized Graphical User Interface (GUI) automation through LLM-powered GUI agents, yet their ability to process sensitive data with limited human oversight raises significant privacy and security risks. This position paper identifies three key risks of GUI agents and examines how they differ from traditional GUI automation and general autonomous agents. Despite these risks, existing evaluations focus primarily on performance, leaving privacy and security assessments largely unexplored. We review current evaluation metrics for both GUI and general LLM agents and outline five key challenges in integrating human evaluators for GUI agent assessments. To address these gaps, we advocate for a human-centered evaluation framework that incorporates risk assessments, enhances user awareness through in-context consent, and embeds privacy and security considerations into GUI agent design and evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_17934 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents Chen, Chaoran Zhang, Zhiping Khalilov, Ibrahim Guo, Bingcan Gebreegziabher, Simret A Ye, Yanfang Xiao, Ziang Yao, Yaxing Li, Tianshi Li, Toby Jia-Jun Human-Computer Interaction Computation and Language Cryptography and Security The rise of Large Language Models (LLMs) has revolutionized Graphical User Interface (GUI) automation through LLM-powered GUI agents, yet their ability to process sensitive data with limited human oversight raises significant privacy and security risks. This position paper identifies three key risks of GUI agents and examines how they differ from traditional GUI automation and general autonomous agents. Despite these risks, existing evaluations focus primarily on performance, leaving privacy and security assessments largely unexplored. We review current evaluation metrics for both GUI and general LLM agents and outline five key challenges in integrating human evaluators for GUI agent assessments. To address these gaps, we advocate for a human-centered evaluation framework that incorporates risk assessments, enhances user awareness through in-context consent, and embeds privacy and security considerations into GUI agent design and evaluation. |
| title | Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents |
| topic | Human-Computer Interaction Computation and Language Cryptography and Security |
| url | https://arxiv.org/abs/2504.17934 |