Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.03146 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911342357643264 |
|---|---|
| author | Zhang, Kaiyuan Yang, Chenghao Wen, Zhoufutu Yuan, Sihang Wang, Qiuyue Huang, Chaoyi Zhu, Guosheng Wang, He Lu, Huawenyu Wen, Jianing Jiao, Jianpeng Luo, Lishu Liu, Longxiang Wu, Sijin Zhu, Xiaolei Zhang, Xuanliang Liu, Yu Zhang, Ge Lin, Yi Shi, Guang Fu, Chaoyou Huang, Wenhao |
| author_facet | Zhang, Kaiyuan Yang, Chenghao Wen, Zhoufutu Yuan, Sihang Wang, Qiuyue Huang, Chaoyi Zhu, Guosheng Wang, He Lu, Huawenyu Wen, Jianing Jiao, Jianpeng Luo, Lishu Liu, Longxiang Wu, Sijin Zhu, Xiaolei Zhang, Xuanliang Liu, Yu Zhang, Ge Lin, Yi Shi, Guang Fu, Chaoyou Huang, Wenhao |
| contents | As reasoning models scale rapidly, the essential role of multimodality in human cognition has come into sharp relief, driving a growing need to probe vision-centric cognitive behaviors. Yet, existing multimodal benchmarks either overemphasize textual reasoning or fall short of systematically capturing vision-centric cognitive behaviors, leaving the cognitive capacity of MLLMs insufficiently assessed. To address this limitation, we introduce MME-CC (Multi-Modal Evaluation benchmark of Cognitive Capacity), a vision-grounded benchmark that organizes 11 representative reasoning tasks into three fundamental categories of visual information: spatial, geometric, and knowledge-based reasoning, and provides fine-grained analyses of MLLMs' cognitive capacity across these dimensions. Based on MME-CC, we conduct extensive experiments over 16 representative MLLMs. Our study reveals that closed-source models currently lead overall (e.g., 42.66 for Gemini-2.5-Pro vs. 30.45 for GLM-4.5V), while spatial and geometric reasoning remain broadly weak (less than or equal to 30%). We further identify common error patterns, including orientation mistakes, fragile cross-view identity persistence, and poor adherence to counterfactual instructions, and observe that Chain-of-Thought typically follows a three-stage process (extract -> reason -> verify) with heavy reliance on visual extraction. We hope this work catalyzes a shift toward treating the cognitive capacity of MLLMs as central to both evaluation and model design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_03146 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MME-CC: A Challenging Multi-Modal Evaluation Benchmark of Cognitive Capacity Zhang, Kaiyuan Yang, Chenghao Wen, Zhoufutu Yuan, Sihang Wang, Qiuyue Huang, Chaoyi Zhu, Guosheng Wang, He Lu, Huawenyu Wen, Jianing Jiao, Jianpeng Luo, Lishu Liu, Longxiang Wu, Sijin Zhu, Xiaolei Zhang, Xuanliang Liu, Yu Zhang, Ge Lin, Yi Shi, Guang Fu, Chaoyou Huang, Wenhao Computation and Language As reasoning models scale rapidly, the essential role of multimodality in human cognition has come into sharp relief, driving a growing need to probe vision-centric cognitive behaviors. Yet, existing multimodal benchmarks either overemphasize textual reasoning or fall short of systematically capturing vision-centric cognitive behaviors, leaving the cognitive capacity of MLLMs insufficiently assessed. To address this limitation, we introduce MME-CC (Multi-Modal Evaluation benchmark of Cognitive Capacity), a vision-grounded benchmark that organizes 11 representative reasoning tasks into three fundamental categories of visual information: spatial, geometric, and knowledge-based reasoning, and provides fine-grained analyses of MLLMs' cognitive capacity across these dimensions. Based on MME-CC, we conduct extensive experiments over 16 representative MLLMs. Our study reveals that closed-source models currently lead overall (e.g., 42.66 for Gemini-2.5-Pro vs. 30.45 for GLM-4.5V), while spatial and geometric reasoning remain broadly weak (less than or equal to 30%). We further identify common error patterns, including orientation mistakes, fragile cross-view identity persistence, and poor adherence to counterfactual instructions, and observe that Chain-of-Thought typically follows a three-stage process (extract -> reason -> verify) with heavy reliance on visual extraction. We hope this work catalyzes a shift toward treating the cognitive capacity of MLLMs as central to both evaluation and model design. |
| title | MME-CC: A Challenging Multi-Modal Evaluation Benchmark of Cognitive Capacity |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.03146 |