VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916354063335424 |
|---|---|
| author | Liu, Xiao Zhang, Tianjie Gu, Yu Iong, Iat Long Xu, Yifan Song, Xixuan Zhang, Shudan Lai, Hanyu Liu, Xinyi Zhao, Hanlin Sun, Jiadai Yang, Xinyue Yang, Yu Qi, Zehan Yao, Shuntian Sun, Xueqiao Cheng, Siyi Zheng, Qinkai Yu, Hao Zhang, Hanchen Hong, Wenyi Ding, Ming Pan, Lihang Gu, Xiaotao Zeng, Aohan Du, Zhengxiao Song, Chan Hee Su, Yu Dong, Yuxiao Tang, Jie |
| author_facet | Liu, Xiao Zhang, Tianjie Gu, Yu Iong, Iat Long Xu, Yifan Song, Xixuan Zhang, Shudan Lai, Hanyu Liu, Xinyi Zhao, Hanlin Sun, Jiadai Yang, Xinyue Yang, Yu Qi, Zehan Yao, Shuntian Sun, Xueqiao Cheng, Siyi Zheng, Qinkai Yu, Hao Zhang, Hanchen Hong, Wenyi Ding, Ming Pan, Lihang Gu, Xiaotao Zeng, Aohan Du, Zhengxiao Song, Chan Hee Su, Yu Dong, Yuxiao Tang, Jie |
| contents | Large Multimodal Models (LMMs) have ushered in a new era in artificial intelligence, merging capabilities in both language and vision to form highly capable Visual Foundation Agents. These agents are postulated to excel across a myriad of tasks, potentially approaching general artificial intelligence. However, existing benchmarks fail to sufficiently challenge or showcase the full potential of LMMs in complex, real-world environments. To address this gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering benchmark specifically designed to train and evaluate LMMs as visual foundation agents across diverse scenarios, including Embodied, Graphical User Interface, and Visual Design, with tasks formulated to probe the depth of LMMs' understanding and interaction capabilities. Through rigorous testing across nine proprietary LMM APIs and eight open models, we demonstrate the considerable yet still developing agent capabilities of these models. Additionally, VAB constructs a trajectory training set constructed through hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and Human Demonstrations, promoting substantial performance improvements in LMMs through behavior cloning. Our work not only aims to benchmark existing models but also provides a solid foundation for future development into visual foundation agents. Code, train \& test data, and part of fine-tuned open LMMs are available at \url{https://github.com/THUDM/VisualAgentBench}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_06327 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents Liu, Xiao Zhang, Tianjie Gu, Yu Iong, Iat Long Xu, Yifan Song, Xixuan Zhang, Shudan Lai, Hanyu Liu, Xinyi Zhao, Hanlin Sun, Jiadai Yang, Xinyue Yang, Yu Qi, Zehan Yao, Shuntian Sun, Xueqiao Cheng, Siyi Zheng, Qinkai Yu, Hao Zhang, Hanchen Hong, Wenyi Ding, Ming Pan, Lihang Gu, Xiaotao Zeng, Aohan Du, Zhengxiao Song, Chan Hee Su, Yu Dong, Yuxiao Tang, Jie Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Large Multimodal Models (LMMs) have ushered in a new era in artificial intelligence, merging capabilities in both language and vision to form highly capable Visual Foundation Agents. These agents are postulated to excel across a myriad of tasks, potentially approaching general artificial intelligence. However, existing benchmarks fail to sufficiently challenge or showcase the full potential of LMMs in complex, real-world environments. To address this gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering benchmark specifically designed to train and evaluate LMMs as visual foundation agents across diverse scenarios, including Embodied, Graphical User Interface, and Visual Design, with tasks formulated to probe the depth of LMMs' understanding and interaction capabilities. Through rigorous testing across nine proprietary LMM APIs and eight open models, we demonstrate the considerable yet still developing agent capabilities of these models. Additionally, VAB constructs a trajectory training set constructed through hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and Human Demonstrations, promoting substantial performance improvements in LMMs through behavior cloning. Our work not only aims to benchmark existing models but also provides a solid foundation for future development into visual foundation agents. Code, train \& test data, and part of fine-tuned open LMMs are available at \url{https://github.com/THUDM/VisualAgentBench}. |
| title | VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.06327 |