MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilot
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917965500252160 |
|---|---|
| author | Song, Zirui Li, Yaohang Fang, Meng Li, Yanda Chen, Zhenhao Shi, Zecheng Huang, Yuan Chen, Xiuying Chen, Ling |
| author_facet | Song, Zirui Li, Yaohang Fang, Meng Li, Yanda Chen, Zhenhao Shi, Zecheng Huang, Yuan Chen, Xiuying Chen, Ling |
| contents | Large language model agents that interact with PC applications often face limitations due to their singular mode of interaction with real-world environments, leading to restricted versatility and frequent hallucinations. To address this, we propose the Multi-Modal Agent Collaboration framework (MMAC-Copilot), a framework utilizes the collective expertise of diverse agents to enhance interaction ability with application. The framework introduces a team collaboration chain, enabling each participating agent to contribute insights based on their specific domain knowledge, effectively reducing the hallucination associated with knowledge domain gaps. We evaluate MMAC-Copilot using the GAIA benchmark and our newly introduced Visual Interaction Benchmark (VIBench). MMAC-Copilot achieved exceptional performance on GAIA, with an average improvement of 6.8\% over existing leading systems. VIBench focuses on non-API-interactable applications across various domains, including 3D gaming, recreation, and office scenarios. It also demonstrated remarkable capability on VIBench. We hope this work can inspire in this field and provide a more comprehensive assessment of Autonomous agents. The anonymous Github is available at \href{https://anonymous.4open.science/r/ComputerAgentWithVision-3C12}{Anonymous Github} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_18074 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilot Song, Zirui Li, Yaohang Fang, Meng Li, Yanda Chen, Zhenhao Shi, Zecheng Huang, Yuan Chen, Xiuying Chen, Ling Artificial Intelligence Human-Computer Interaction Large language model agents that interact with PC applications often face limitations due to their singular mode of interaction with real-world environments, leading to restricted versatility and frequent hallucinations. To address this, we propose the Multi-Modal Agent Collaboration framework (MMAC-Copilot), a framework utilizes the collective expertise of diverse agents to enhance interaction ability with application. The framework introduces a team collaboration chain, enabling each participating agent to contribute insights based on their specific domain knowledge, effectively reducing the hallucination associated with knowledge domain gaps. We evaluate MMAC-Copilot using the GAIA benchmark and our newly introduced Visual Interaction Benchmark (VIBench). MMAC-Copilot achieved exceptional performance on GAIA, with an average improvement of 6.8\% over existing leading systems. VIBench focuses on non-API-interactable applications across various domains, including 3D gaming, recreation, and office scenarios. It also demonstrated remarkable capability on VIBench. We hope this work can inspire in this field and provide a more comprehensive assessment of Autonomous agents. The anonymous Github is available at \href{https://anonymous.4open.science/r/ComputerAgentWithVision-3C12}{Anonymous Github} |
| title | MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilot |
| topic | Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2404.18074 |