Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908999540015104 |
|---|---|
| author | Gao, Alice Tong, Weixi Vempati, Rishab Reinecke, Katharina Shapiro, R. Benjamin Zhang, Tianyi Wu, Jason |
| author_facet | Gao, Alice Tong, Weixi Vempati, Rishab Reinecke, Katharina Shapiro, R. Benjamin Zhang, Tianyi Wu, Jason |
| contents | Usability testing with experts and potential users can assess the effectiveness, efficiency, and user satisfaction of graphical user interfaces (GUIs) but doing so remains a costly and time-intensive process. Prior work has used computer use agents (CUAs) and other generative agents that can simulate user interactions and preference, but we show that agents still struggle to provide accurate usability assessments. In this work, we present a novel machine learning method that operationalizes a computational definition of usability to train CUAs to assess GUI usability by i) prioritizing important interaction flows, ii) executing them through human-like interactions, and iii) predicting a learned numerical usability score. We train a computer use agent, uxCUA, with our algorithm on a large-scale dataset of fully interactive user interfaces (UIs) paired with usability labels and human preferences. We show that uxCUA outperforms larger models in accurate usability assessments and produces realistic critiques of both synthetic and real UIs. More broadly, our work aims to build a principled, data-driven foundation for automated usability assessment in HCI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_26020 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Training Computer Use Agents to Assess the Usability of Graphical User Interfaces Gao, Alice Tong, Weixi Vempati, Rishab Reinecke, Katharina Shapiro, R. Benjamin Zhang, Tianyi Wu, Jason Computation and Language Artificial Intelligence Usability testing with experts and potential users can assess the effectiveness, efficiency, and user satisfaction of graphical user interfaces (GUIs) but doing so remains a costly and time-intensive process. Prior work has used computer use agents (CUAs) and other generative agents that can simulate user interactions and preference, but we show that agents still struggle to provide accurate usability assessments. In this work, we present a novel machine learning method that operationalizes a computational definition of usability to train CUAs to assess GUI usability by i) prioritizing important interaction flows, ii) executing them through human-like interactions, and iii) predicting a learned numerical usability score. We train a computer use agent, uxCUA, with our algorithm on a large-scale dataset of fully interactive user interfaces (UIs) paired with usability labels and human preferences. We show that uxCUA outperforms larger models in accurate usability assessments and produces realistic critiques of both synthetic and real UIs. More broadly, our work aims to build a principled, data-driven foundation for automated usability assessment in HCI. |
| title | Training Computer Use Agents to Assess the Usability of Graphical User Interfaces |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.26020 |