Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915974576340992 |
|---|---|
| author | Huang, Jen-Tse Dai, Dasen Huang, Jen-Yuan Yuan, Youliang Liu, Xiaoyuan Wang, Wenxuan Jiao, Wenxiang He, Pinjia Tu, Zhaopeng Duan, Haodong |
| author_facet | Huang, Jen-Tse Dai, Dasen Huang, Jen-Yuan Yuan, Youliang Liu, Xiaoyuan Wang, Wenxuan Jiao, Wenxiang He, Pinjia Tu, Zhaopeng Duan, Haodong |
| contents | Humans develop perception through a bottom-up hierarchy: from basic primitives and Gestalt principles to high-level semantics. In contrast, current Multimodal Large Language Models (MLLMs) are trained directly on complex downstream tasks, often bypassing these foundational visual capabilities. To systematically investigate this gap, we introduce VisFactor, a benchmark that digitizes 20 vision-centric subtests from FRCT, a well-established cognitive psychology assessment spanning four domains of human visual cognition. Furthermore, we design algorithms to automatically construct and validate unlimited test cases with controllable difficulty. Using VisFactor, we evaluate 39 frontier MLLMs, including both proprietary (e.g., GPT, Gemini) and open-source (e.g., LLaMA, Qwen) models. The best model achieves a score of only 54.0%. Analysis reveals good internal consistency (Cronbach's alpha = 0.94) and construct validity (compared to existing vision benchmarks). Models consistently fail on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination, regardless of model size or prompting strategy. These findings suggest that performance improvements on existing general benchmarks might represent castles in the air instead of a genuine mastery of human-like visual cognition. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16435 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs Huang, Jen-Tse Dai, Dasen Huang, Jen-Yuan Yuan, Youliang Liu, Xiaoyuan Wang, Wenxuan Jiao, Wenxiang He, Pinjia Tu, Zhaopeng Duan, Haodong Computer Vision and Pattern Recognition Computation and Language Humans develop perception through a bottom-up hierarchy: from basic primitives and Gestalt principles to high-level semantics. In contrast, current Multimodal Large Language Models (MLLMs) are trained directly on complex downstream tasks, often bypassing these foundational visual capabilities. To systematically investigate this gap, we introduce VisFactor, a benchmark that digitizes 20 vision-centric subtests from FRCT, a well-established cognitive psychology assessment spanning four domains of human visual cognition. Furthermore, we design algorithms to automatically construct and validate unlimited test cases with controllable difficulty. Using VisFactor, we evaluate 39 frontier MLLMs, including both proprietary (e.g., GPT, Gemini) and open-source (e.g., LLaMA, Qwen) models. The best model achieves a score of only 54.0%. Analysis reveals good internal consistency (Cronbach's alpha = 0.94) and construct validity (compared to existing vision benchmarks). Models consistently fail on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination, regardless of model size or prompting strategy. These findings suggest that performance improvements on existing general benchmarks might represent castles in the air instead of a genuine mastery of human-like visual cognition. |
| title | Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2502.16435 |