Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.22137 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908560939548672 |
|---|---|
| author | Lee, Seoyoung Yoon, Seonbin Lee, Seongbeen Kim, Hyesoo Sim, Joo Yong |
| author_facet | Lee, Seoyoung Yoon, Seonbin Lee, Seongbeen Kim, Hyesoo Sim, Joo Yong |
| contents | GUI task automation streamlines repetitive tasks, but existing LLM or VLM-based planner-executor agents suffer from brittle generalization, high latency, and limited long-horizon coherence. Their reliance on single-shot reasoning or static plans makes them fragile under UI changes or complex tasks. Log2Plan addresses these limitations by combining a structured two-level planning framework with a task mining approach over user behavior logs, enabling robust and adaptable GUI automation. Log2Plan constructs high-level plans by mapping user commands to a structured task dictionary, enabling consistent and generalizable automation. To support personalization and reuse, it employs a task mining approach from user behavior logs that identifies user-specific patterns. These high-level plans are then grounded into low-level action sequences by interpreting real-time GUI context, ensuring robust execution across varying interfaces. We evaluated Log2Plan on 200 real-world tasks, demonstrating significant improvements in task success rate and execution time. Notably, it maintains over 60.0% success rate even on long-horizon task sequences, highlighting its robustness in complex, multi-step workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_22137 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining Approach Lee, Seoyoung Yoon, Seonbin Lee, Seongbeen Kim, Hyesoo Sim, Joo Yong Artificial Intelligence Human-Computer Interaction Multiagent Systems Robotics 68N19, 68T09 H.5.2; D.2.2 GUI task automation streamlines repetitive tasks, but existing LLM or VLM-based planner-executor agents suffer from brittle generalization, high latency, and limited long-horizon coherence. Their reliance on single-shot reasoning or static plans makes them fragile under UI changes or complex tasks. Log2Plan addresses these limitations by combining a structured two-level planning framework with a task mining approach over user behavior logs, enabling robust and adaptable GUI automation. Log2Plan constructs high-level plans by mapping user commands to a structured task dictionary, enabling consistent and generalizable automation. To support personalization and reuse, it employs a task mining approach from user behavior logs that identifies user-specific patterns. These high-level plans are then grounded into low-level action sequences by interpreting real-time GUI context, ensuring robust execution across varying interfaces. We evaluated Log2Plan on 200 real-world tasks, demonstrating significant improvements in task success rate and execution time. Notably, it maintains over 60.0% success rate even on long-horizon task sequences, highlighting its robustness in complex, multi-step workflows. |
| title | Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining Approach |
| topic | Artificial Intelligence Human-Computer Interaction Multiagent Systems Robotics 68N19, 68T09 H.5.2; D.2.2 |
| url | https://arxiv.org/abs/2509.22137 |