META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910348820348928 |
|---|---|
| author | Sun, Liangtai Chen, Xingyu Chen, Lu Dai, Tianle Zhu, Zichen Yu, Kai |
| author_facet | Sun, Liangtai Chen, Xingyu Chen, Lu Dai, Tianle Zhu, Zichen Yu, Kai |
| contents | Task-oriented dialogue (TOD) systems have been widely used by mobile phone intelligent assistants to accomplish tasks such as calendar scheduling or hotel reservation. Current TOD systems usually focus on multi-turn text/speech interaction, then they would call back-end APIs designed for TODs to perform the task. However, this API-based architecture greatly limits the information-searching capability of intelligent assistants and may even lead to task failure if TOD-specific APIs are not available or the task is too complicated to be executed by the provided APIs. In this paper, we propose a new TOD architecture: GUI-based task-oriented dialogue system (GUI-TOD). A GUI-TOD system can directly perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs. Furthermore, we release META-GUI, a dataset for training a Multi-modal convErsaTional Agent on mobile GUI. We also propose a multi-model action prediction and response model, which show promising results on META-GUI. The dataset, codes and leaderboard are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2205_11029 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI Sun, Liangtai Chen, Xingyu Chen, Lu Dai, Tianle Zhu, Zichen Yu, Kai Computation and Language Artificial Intelligence Task-oriented dialogue (TOD) systems have been widely used by mobile phone intelligent assistants to accomplish tasks such as calendar scheduling or hotel reservation. Current TOD systems usually focus on multi-turn text/speech interaction, then they would call back-end APIs designed for TODs to perform the task. However, this API-based architecture greatly limits the information-searching capability of intelligent assistants and may even lead to task failure if TOD-specific APIs are not available or the task is too complicated to be executed by the provided APIs. In this paper, we propose a new TOD architecture: GUI-based task-oriented dialogue system (GUI-TOD). A GUI-TOD system can directly perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs. Furthermore, we release META-GUI, a dataset for training a Multi-modal convErsaTional Agent on mobile GUI. We also propose a multi-model action prediction and response model, which show promising results on META-GUI. The dataset, codes and leaderboard are publicly available. |
| title | META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2205.11029 |