Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.07023 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910411545116672 |
|---|---|
| author | Huang, Forrest Li, Gang Li, Tao Li, Yang |
| author_facet | Huang, Forrest Li, Gang Li, Tao Li, Yang |
| contents | Macros are building block tasks of our everyday smartphone activity (e.g., "login", or "booking a flight"). Effectively extracting macros is important for understanding mobile interaction and enabling task automation. These macros are however difficult to extract at scale as they can be comprised of multiple steps yet hidden within programmatic components of mobile apps. In this paper, we introduce a novel approach based on Large Language Models (LLMs) to automatically extract semantically meaningful macros from both random and user-curated mobile interaction traces. The macros produced by our approach are automatically tagged with natural language descriptions and are fully executable. We conduct multiple studies to validate the quality of extracted macros, including user evaluation, comparative analysis against human-curated tasks, and automatic execution of these macros. These experiments and analyses show the effectiveness of our approach and the usefulness of extracted macros in various downstream applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_07023 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Automatic Macro Mining from Interaction Traces at Scale Huang, Forrest Li, Gang Li, Tao Li, Yang Human-Computer Interaction Computation and Language Machine Learning Macros are building block tasks of our everyday smartphone activity (e.g., "login", or "booking a flight"). Effectively extracting macros is important for understanding mobile interaction and enabling task automation. These macros are however difficult to extract at scale as they can be comprised of multiple steps yet hidden within programmatic components of mobile apps. In this paper, we introduce a novel approach based on Large Language Models (LLMs) to automatically extract semantically meaningful macros from both random and user-curated mobile interaction traces. The macros produced by our approach are automatically tagged with natural language descriptions and are fully executable. We conduct multiple studies to validate the quality of extracted macros, including user evaluation, comparative analysis against human-curated tasks, and automatic execution of these macros. These experiments and analyses show the effectiveness of our approach and the usefulness of extracted macros in various downstream applications. |
| title | Automatic Macro Mining from Interaction Traces at Scale |
| topic | Human-Computer Interaction Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2310.07023 |