Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915202296971264 |
|---|---|
| author | Huang, Tian Yu, Chun Shi, Weinan Peng, Zijian Yang, David Sun, Weiqi Shi, Yuanchun |
| author_facet | Huang, Tian Yu, Chun Shi, Weinan Peng, Zijian Yang, David Sun, Weiqi Shi, Yuanchun |
| contents | UI task automation enables efficient task execution by simulating human interactions with graphical user interfaces (GUIs), without modifying the existing application code. However, its broader adoption is constrained by the need for expertise in both scripting languages and workflow design. To address this challenge, we present Prompt2Task, a system designed to comprehend various task-related textual prompts (e.g., goals, procedures), thereby generating and performing the corresponding automation tasks. Prompt2Task incorporates a suite of intelligent agents that mimic human cognitive functions, specializing in interpreting user intent, managing external information for task generation, and executing operations on smartphones. The agents can learn from user feedback and continuously improve their performance based on the accumulated knowledge. Experimental results indicated a performance jump from a 22.28\% success rate in the baseline to 95.24\% with Prompt2Task, requiring an average of 0.69 user interventions for each new task. Prompt2Task presents promising applications in fields such as tutorial creation, smart assistance, and customer service. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_02475 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts Huang, Tian Yu, Chun Shi, Weinan Peng, Zijian Yang, David Sun, Weiqi Shi, Yuanchun Human-Computer Interaction UI task automation enables efficient task execution by simulating human interactions with graphical user interfaces (GUIs), without modifying the existing application code. However, its broader adoption is constrained by the need for expertise in both scripting languages and workflow design. To address this challenge, we present Prompt2Task, a system designed to comprehend various task-related textual prompts (e.g., goals, procedures), thereby generating and performing the corresponding automation tasks. Prompt2Task incorporates a suite of intelligent agents that mimic human cognitive functions, specializing in interpreting user intent, managing external information for task generation, and executing operations on smartphones. The agents can learn from user feedback and continuously improve their performance based on the accumulated knowledge. Experimental results indicated a performance jump from a 22.28\% success rate in the baseline to 95.24\% with Prompt2Task, requiring an average of 0.69 user interventions for each new task. Prompt2Task presents promising applications in fields such as tutorial creation, smart assistance, and customer service. |
| title | Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2404.02475 |