Comparing Human Oversight Strategies for Computer-Use Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908940163350528 |
|---|---|
| author | Chen, Chaoran Zhang, Zhiping Chen, Zeya Xu, Eryue Yang, Yinuo Khalilov, Ibrahim Gebreegziabher, Simret A Ye, Yanfang Xiao, Ziang Yao, Yaxing Li, Tianshi Li, Toby Jia-Jun |
| author_facet | Chen, Chaoran Zhang, Zhiping Chen, Zeya Xu, Eryue Yang, Yinuo Khalilov, Ibrahim Gebreegziabher, Simret A Ye, Yanfang Xiao, Ziang Yao, Yaxing Li, Tianshi Li, Toby Jia-Jun |
| contents | LLM-powered computer-use agents (CUAs) are shifting users from direct manipulation to supervisory coordination. Existing oversight mechanisms, however, have largely been studied as isolated interface features, making broader oversight strategies difficult to compare. We conceptualize CUA oversight as a structural coordination problem defined by delegation structure and engagement level, and use this lens to compare four oversight strategies in a mixed-methods study with 48 participants in a live web environment. Our results show that oversight strategy more reliably shaped users' exposure to problematic actions than their ability to correct them once visible. Plan-based strategies were associated with lower rates of agent problematic-action occurrence, but not equally strong gains in runtime intervention success once such actions became visible. On subjective measures, no single strategy was uniformly best, and the clearest context-sensitive differences appeared in trust. Qualitative findings further suggest that intervention depended not only on what controls users retained, but on whether risky moments became legible as requiring judgment during execution. These findings suggest that effective CUA oversight is not achieved by maximizing human involvement alone. Instead, it depends on how supervision is structured to surface decision-critical moments and support their recognition in time for meaningful intervention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_04918 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Comparing Human Oversight Strategies for Computer-Use Agents Chen, Chaoran Zhang, Zhiping Chen, Zeya Xu, Eryue Yang, Yinuo Khalilov, Ibrahim Gebreegziabher, Simret A Ye, Yanfang Xiao, Ziang Yao, Yaxing Li, Tianshi Li, Toby Jia-Jun Human-Computer Interaction LLM-powered computer-use agents (CUAs) are shifting users from direct manipulation to supervisory coordination. Existing oversight mechanisms, however, have largely been studied as isolated interface features, making broader oversight strategies difficult to compare. We conceptualize CUA oversight as a structural coordination problem defined by delegation structure and engagement level, and use this lens to compare four oversight strategies in a mixed-methods study with 48 participants in a live web environment. Our results show that oversight strategy more reliably shaped users' exposure to problematic actions than their ability to correct them once visible. Plan-based strategies were associated with lower rates of agent problematic-action occurrence, but not equally strong gains in runtime intervention success once such actions became visible. On subjective measures, no single strategy was uniformly best, and the clearest context-sensitive differences appeared in trust. Qualitative findings further suggest that intervention depended not only on what controls users retained, but on whether risky moments became legible as requiring judgment during execution. These findings suggest that effective CUA oversight is not achieved by maximizing human involvement alone. Instead, it depends on how supervision is structured to surface decision-critical moments and support their recognition in time for meaningful intervention. |
| title | Comparing Human Oversight Strategies for Computer-Use Agents |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2604.04918 |