Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917458911166464 |
|---|---|
| author | Liu, Jiateng Wang, Rushi Li, Bingxuan Zhu, Kunlun Shen, Yifan Wang, Qingyun Abbasi, Ahmed Zhang, Denghui Ji, Heng |
| author_facet | Liu, Jiateng Wang, Rushi Li, Bingxuan Zhu, Kunlun Shen, Yifan Wang, Qingyun Abbasi, Ahmed Zhang, Denghui Ji, Heng |
| contents | Recent advances have enabled general computer-use agents that interpret screens and execute grounded actions from human instructions, yet they still struggle to generalize to unseen and evolving interfaces. While improving agent capability remains important, agent compatible interface design offers a complementary path by aligning interaction semantics with agent prior knowledge. In this paper, we revisit Nielsen 10 usability heuristics through the lens of computer-use agents, identifying which principles naturally transfer, where implicit design assumptions create agent specific failures, and how safe additive augmentations can improve robustness without harming human usability. To evaluate these ideas, we introduce UI-Verse, a suite of controlled environments built around functionally similar interfaces with different applied heuristics. Experiments show that our augmented heuristics consistently improve task completion and modestly improve efficiency, with combined heuristics yielding further gains. Human studies further show that these designs preserve the original interaction workflow without observable usability regressions. Overall, our findings highlight interface design as a practical complementary avenue for improving the reliability and generalization of computer use agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_02729 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents Liu, Jiateng Wang, Rushi Li, Bingxuan Zhu, Kunlun Shen, Yifan Wang, Qingyun Abbasi, Ahmed Zhang, Denghui Ji, Heng Human-Computer Interaction Recent advances have enabled general computer-use agents that interpret screens and execute grounded actions from human instructions, yet they still struggle to generalize to unseen and evolving interfaces. While improving agent capability remains important, agent compatible interface design offers a complementary path by aligning interaction semantics with agent prior knowledge. In this paper, we revisit Nielsen 10 usability heuristics through the lens of computer-use agents, identifying which principles naturally transfer, where implicit design assumptions create agent specific failures, and how safe additive augmentations can improve robustness without harming human usability. To evaluate these ideas, we introduce UI-Verse, a suite of controlled environments built around functionally similar interfaces with different applied heuristics. Experiments show that our augmented heuristics consistently improve task completion and modestly improve efficiency, with combined heuristics yielding further gains. Human studies further show that these designs preserve the original interaction workflow without observable usability regressions. Overall, our findings highlight interface design as a practical complementary avenue for improving the reliability and generalization of computer use agents. |
| title | Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2605.02729 |