Holistic Evaluation of Multimodal LLMs on Spatial Intelligence
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912796306833408 |
|---|---|
| author | Cai, Zhongang Wang, Yubo Sun, Qingping Wang, Ruisi Gu, Chenyang Yin, Wanqi Lin, Zhiqian Yang, Zhitao Wei, Chen Qian, Oscar Pang, Hui En Shi, Xuanke Deng, Kewang Han, Xiaoyang Chen, Zukai Li, Jiaqi Fan, Xiangyu Deng, Hanming Lu, Lewei Li, Bo Liu, Ziwei Wang, Quan Lin, Dahua Yang, Lei |
| author_facet | Cai, Zhongang Wang, Yubo Sun, Qingping Wang, Ruisi Gu, Chenyang Yin, Wanqi Lin, Zhiqian Yang, Zhitao Wei, Chen Qian, Oscar Pang, Hui En Shi, Xuanke Deng, Kewang Han, Xiaoyang Chen, Zukai Li, Jiaqi Fan, Xiangyu Deng, Hanming Lu, Lewei Li, Bo Liu, Ziwei Wang, Quan Lin, Dahua Yang, Lei |
| contents | Multimodal models have achieved remarkable progress in recent years. Nevertheless, they continue to exhibit notable limitations in spatial understanding and reasoning, the very capability that anchors artificial general intelligence in the physical world. With the recent release of GPT-5, allegedly the most powerful AI model to date, it is timely to examine where the leading models (GPT, Gemini, Grok, Seed, Qwen, and Intern) stand on the path toward spatial intelligence (SI). We thus propose EASI for holistic Evaluation of multimodAl LLMs on Spatial Intelligence. EASI conceptualizes a comprehensive taxonomy of spatial tasks that unifies existing benchmarks and a growing collection of newly curated ones, enabling systematic evaluation of state-of-the-art models. In this report, we conduct the study across eight key benchmarks, at a cost exceeding ten billion total tokens. Our empirical study then reveals that (1) GPT-5 demonstrates unprecedented strength in SI, yet (2) still falls short of human performance significantly across a broad spectrum of SI-tasks. Moreover, we (3) show that SI-tasks expose greater model capability deficiency than non-SI tasks, to the extent that (4) proprietary models do not exhibit a decisive advantage when facing the most difficult ones. In addition, we conduct a qualitative evaluation across a diverse set of scenarios that are intuitive for humans, yet fail the most advanced multimodal models. EASI is an ongoing community effort: we have open-sourced the EASI codebase that provides a one-stop and reproducible solution with standardized interfaces, integrated protocols and prompts that significantly reduce the friction of configuring and running multiple benchmarks; we have also launched an accompanying EASI leaderboard to provide a continually updated snapshot of model performance across the full SI spectrum, accelerating collective progress toward robust SI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_13142 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Holistic Evaluation of Multimodal LLMs on Spatial Intelligence Cai, Zhongang Wang, Yubo Sun, Qingping Wang, Ruisi Gu, Chenyang Yin, Wanqi Lin, Zhiqian Yang, Zhitao Wei, Chen Qian, Oscar Pang, Hui En Shi, Xuanke Deng, Kewang Han, Xiaoyang Chen, Zukai Li, Jiaqi Fan, Xiangyu Deng, Hanming Lu, Lewei Li, Bo Liu, Ziwei Wang, Quan Lin, Dahua Yang, Lei Computer Vision and Pattern Recognition Computation and Language Machine Learning Multimedia Robotics Multimodal models have achieved remarkable progress in recent years. Nevertheless, they continue to exhibit notable limitations in spatial understanding and reasoning, the very capability that anchors artificial general intelligence in the physical world. With the recent release of GPT-5, allegedly the most powerful AI model to date, it is timely to examine where the leading models (GPT, Gemini, Grok, Seed, Qwen, and Intern) stand on the path toward spatial intelligence (SI). We thus propose EASI for holistic Evaluation of multimodAl LLMs on Spatial Intelligence. EASI conceptualizes a comprehensive taxonomy of spatial tasks that unifies existing benchmarks and a growing collection of newly curated ones, enabling systematic evaluation of state-of-the-art models. In this report, we conduct the study across eight key benchmarks, at a cost exceeding ten billion total tokens. Our empirical study then reveals that (1) GPT-5 demonstrates unprecedented strength in SI, yet (2) still falls short of human performance significantly across a broad spectrum of SI-tasks. Moreover, we (3) show that SI-tasks expose greater model capability deficiency than non-SI tasks, to the extent that (4) proprietary models do not exhibit a decisive advantage when facing the most difficult ones. In addition, we conduct a qualitative evaluation across a diverse set of scenarios that are intuitive for humans, yet fail the most advanced multimodal models. EASI is an ongoing community effort: we have open-sourced the EASI codebase that provides a one-stop and reproducible solution with standardized interfaces, integrated protocols and prompts that significantly reduce the friction of configuring and running multiple benchmarks; we have also launched an accompanying EASI leaderboard to provide a continually updated snapshot of model performance across the full SI spectrum, accelerating collective progress toward robust SI. |
| title | Holistic Evaluation of Multimodal LLMs on Spatial Intelligence |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning Multimedia Robotics |
| url | https://arxiv.org/abs/2508.13142 |