Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2407.01511 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909695160090624 |
|---|---|
| author | Xu, Tianqi Chen, Linyao Wu, Dai-Jie Chen, Yanjun Zhang, Zecheng Yao, Xiang Xie, Zhiqiang Chen, Yongchao Liu, Shilong Qian, Bochen Yang, Anjie Jin, Zhaoxuan Deng, Jianbo Torr, Philip Ghanem, Bernard Li, Guohao |
| author_facet | Xu, Tianqi Chen, Linyao Wu, Dai-Jie Chen, Yanjun Zhang, Zecheng Yao, Xiang Xie, Zhiqiang Chen, Yongchao Liu, Shilong Qian, Bochen Yang, Anjie Jin, Zhaoxuan Deng, Jianbo Torr, Philip Ghanem, Bernard Li, Guohao |
| contents | The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks for MLM agents in interactive environments are limited by their focus on a single environment, lack of detailed and generalized evaluation methods, and the complexities of constructing tasks and evaluators. To overcome these limitations, we introduce Crab, the first agent benchmark framework designed to support cross-environment tasks, incorporating a graph-based fine-grained evaluation method and an efficient mechanism for task and evaluator construction. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. Leveraging Crab, we developed a cross-platform Crab Benchmark-v0 comprising 120 tasks in computer desktop and mobile phone environments. We evaluated four advanced MLMs using different single and multi-agent system configurations on this benchmark. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%. All framework code, agent code, and task datasets are publicly available at https://github.com/camel-ai/crab. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01511 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents Xu, Tianqi Chen, Linyao Wu, Dai-Jie Chen, Yanjun Zhang, Zecheng Yao, Xiang Xie, Zhiqiang Chen, Yongchao Liu, Shilong Qian, Bochen Yang, Anjie Jin, Zhaoxuan Deng, Jianbo Torr, Philip Ghanem, Bernard Li, Guohao Artificial Intelligence The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks for MLM agents in interactive environments are limited by their focus on a single environment, lack of detailed and generalized evaluation methods, and the complexities of constructing tasks and evaluators. To overcome these limitations, we introduce Crab, the first agent benchmark framework designed to support cross-environment tasks, incorporating a graph-based fine-grained evaluation method and an efficient mechanism for task and evaluator construction. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. Leveraging Crab, we developed a cross-platform Crab Benchmark-v0 comprising 120 tasks in computer desktop and mobile phone environments. We evaluated four advanced MLMs using different single and multi-agent system configurations on this benchmark. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%. All framework code, agent code, and task datasets are publicly available at https://github.com/camel-ai/crab. |
| title | CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2407.01511 |