UINav: A Practical Approach to Train On-Device Automation Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910505152544768 |
|---|---|
| author | Li, Wei Hsu, Fu-Lin Bishop, Will Campbell-Ajala, Folawiyo Lin, Max Riva, Oriana |
| author_facet | Li, Wei Hsu, Fu-Lin Bishop, Will Campbell-Ajala, Folawiyo Lin, Max Riva, Oriana |
| contents | Automation systems that can autonomously drive application user interfaces to complete user tasks are of great benefit, especially when users are situationally or permanently impaired. Prior automation systems do not produce generalizable models while AI-based automation agents work reliably only in simple, hand-crafted applications or incur high computation costs. We propose UINav, a demonstration-based approach to train automation agents that fit mobile devices, yet achieving high success rates with modest numbers of demonstrations. To reduce the demonstration overhead, UINav uses a referee model that provides users with immediate feedback on tasks where the agent fails, and automatically augments human demonstrations to increase diversity in training data. Our evaluation shows that with only 10 demonstrations UINav can achieve 70% accuracy, and that with enough demonstrations it can surpass 90% accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_10170 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | UINav: A Practical Approach to Train On-Device Automation Agents Li, Wei Hsu, Fu-Lin Bishop, Will Campbell-Ajala, Folawiyo Lin, Max Riva, Oriana Human-Computer Interaction Artificial Intelligence Automation systems that can autonomously drive application user interfaces to complete user tasks are of great benefit, especially when users are situationally or permanently impaired. Prior automation systems do not produce generalizable models while AI-based automation agents work reliably only in simple, hand-crafted applications or incur high computation costs. We propose UINav, a demonstration-based approach to train automation agents that fit mobile devices, yet achieving high success rates with modest numbers of demonstrations. To reduce the demonstration overhead, UINav uses a referee model that provides users with immediate feedback on tasks where the agent fails, and automatically augments human demonstrations to increase diversity in training data. Our evaluation shows that with only 10 demonstrations UINav can achieve 70% accuracy, and that with enough demonstrations it can surpass 90% accuracy. |
| title | UINav: A Practical Approach to Train On-Device Automation Agents |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2312.10170 |