TaskSense: Cognitive Chain Modeling and Difficulty Estimation for GUI Tasks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918198774857728 |
|---|---|
| author | Yin, Yiwen Hu, Zhian Xu, Xiaoxi Yu, Chun Wu, Xintong Fan, Wenyu Shi, Yuanchun |
| author_facet | Yin, Yiwen Hu, Zhian Xu, Xiaoxi Yu, Chun Wu, Xintong Fan, Wenyu Shi, Yuanchun |
| contents | Measuring GUI task difficulty is crucial for user behavior analysis and agent capability evaluation. Yet, existing benchmarks typically quantify difficulty based on motor actions (e.g., step counts), overlooking the cognitive demands underlying task completion. In this work, we propose Cognitive Chain, a novel framework that models task difficulty from a cognitive perspective. A cognitive chain decomposes the cognitive processes preceding a motor action into a sequence of cognitive steps (e.g., finding, deciding, computing), each with a difficulty index grounded in information theories. We develop an LLM-based method to automatically extract cognitive chains from task execution traces. Validation with linear regression shows that our estimated cognitive difficulty correlates well with user completion time (step-level R-square=0.46 after annotation). Assessment of state-of-the-art GUI agents shows reduced success on cognitively demanding tasks, revealing capability gaps and Human-AI consistency patterns. We conclude by discussing potential applications in agent training, capability assessment, and human-agent delegation optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_09309 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TaskSense: Cognitive Chain Modeling and Difficulty Estimation for GUI Tasks Yin, Yiwen Hu, Zhian Xu, Xiaoxi Yu, Chun Wu, Xintong Fan, Wenyu Shi, Yuanchun Human-Computer Interaction Artificial Intelligence Measuring GUI task difficulty is crucial for user behavior analysis and agent capability evaluation. Yet, existing benchmarks typically quantify difficulty based on motor actions (e.g., step counts), overlooking the cognitive demands underlying task completion. In this work, we propose Cognitive Chain, a novel framework that models task difficulty from a cognitive perspective. A cognitive chain decomposes the cognitive processes preceding a motor action into a sequence of cognitive steps (e.g., finding, deciding, computing), each with a difficulty index grounded in information theories. We develop an LLM-based method to automatically extract cognitive chains from task execution traces. Validation with linear regression shows that our estimated cognitive difficulty correlates well with user completion time (step-level R-square=0.46 after annotation). Assessment of state-of-the-art GUI agents shows reduced success on cognitively demanding tasks, revealing capability gaps and Human-AI consistency patterns. We conclude by discussing potential applications in agent training, capability assessment, and human-agent delegation optimization. |
| title | TaskSense: Cognitive Chain Modeling and Difficulty Estimation for GUI Tasks |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2511.09309 |