AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912381752311808 |
|---|---|
| author | Lu, Junting Zhang, Zhiyang Yang, Fangkai Zhang, Jue Wang, Lu Du, Chao Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Zhang, Qi |
| author_facet | Lu, Junting Zhang, Zhiyang Yang, Fangkai Zhang, Jue Wang, Lu Du, Chao Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Zhang, Qi |
| contents | Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents' performance in complex tasks. However, these agents often suffer from high latency and low reliability due to the extensive sequential UI interactions. To address this issue, we propose AXIS, a novel LLM-based agents framework that prioritize actions through application programming interfaces (APIs) over UI actions. This framework also facilitates the creation and expansion of APIs through automated exploration of applications. Our experiments on Microsoft Word demonstrate that AXIS reduces task completion time by 65%-70% and cognitive workload by 38%-53%, while maintaining accuracy of 97%-98% compared to humans. Our work contributes to a new human-agent-computer interaction (HACI) framework and explores a fresh UI design principle for application providers to turn applications into agents in the era of LLMs, paving the way towards an agent-centric operating system (Agent OS). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17140 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents Lu, Junting Zhang, Zhiyang Yang, Fangkai Zhang, Jue Wang, Lu Du, Chao Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Zhang, Qi Artificial Intelligence Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents' performance in complex tasks. However, these agents often suffer from high latency and low reliability due to the extensive sequential UI interactions. To address this issue, we propose AXIS, a novel LLM-based agents framework that prioritize actions through application programming interfaces (APIs) over UI actions. This framework also facilitates the creation and expansion of APIs through automated exploration of applications. Our experiments on Microsoft Word demonstrate that AXIS reduces task completion time by 65%-70% and cognitive workload by 38%-53%, while maintaining accuracy of 97%-98% compared to humans. Our work contributes to a new human-agent-computer interaction (HACI) framework and explores a fresh UI design principle for application providers to turn applications into agents in the era of LLMs, paving the way towards an agent-centric operating system (Agent OS). |
| title | AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2409.17140 |