Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.10521 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909901329006592 |
|---|---|
| author | Zhou, Yuhao Wang, Yiheng He, Xuming Shen, Ao Xiao, Ruoyao Li, Zhiwei Feng, Qiantai Guo, Zijie Yang, Yuejin Wu, Hao Huang, Wenxuan Wei, Jiaqi Si, Dan Yao, Xiuqi Bu, Jia Huang, Haiwen Wang, Manning Fu, Tianfan Tang, Shixiang Fei, Ben Zhou, Dongzhan Ling, Fenghua Lu, Yan Sun, Siqi Li, Chenhui Zheng, Guanjie Lv, Jiancheng Zhang, Wenlong Bai, Lei |
| author_facet | Zhou, Yuhao Wang, Yiheng He, Xuming Shen, Ao Xiao, Ruoyao Li, Zhiwei Feng, Qiantai Guo, Zijie Yang, Yuejin Wu, Hao Huang, Wenxuan Wei, Jiaqi Si, Dan Yao, Xiuqi Bu, Jia Huang, Haiwen Wang, Manning Fu, Tianfan Tang, Shixiang Fei, Ben Zhou, Dongzhan Ling, Fenghua Lu, Yan Sun, Siqi Li, Chenhui Zheng, Guanjie Lv, Jiancheng Zhang, Wenlong Bai, Lei |
| contents | Scientific discoveries increasingly rely on complex multimodal reasoning based on information-intensive scientific data and domain-specific expertise. Empowered by expert-level scientific benchmarks, scientific Multimodal Large Language Models (MLLMs) hold the potential to significantly enhance this discovery process in realistic workflows. However, current scientific benchmarks mostly focus on evaluating the knowledge understanding capabilities of MLLMs, leading to an inadequate assessment of their perception and reasoning abilities. To address this gap, we present the Scientists' First Exam (SFE) benchmark, designed to evaluate the scientific cognitive capacities of MLLMs through three interconnected levels: scientific signal perception, scientific attribute understanding, scientific comparative reasoning. Specifically, SFE comprises 830 expert-verified VQA pairs across three question types, spanning 66 multimodal tasks across five high-value disciplines. Extensive experiments reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08% and 26.52% on SFE, highlighting significant room for MLLMs to improve in scientific realms. We hope the insights obtained in SFE will facilitate further developments in AI-enhanced scientific discoveries. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10521 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning Zhou, Yuhao Wang, Yiheng He, Xuming Shen, Ao Xiao, Ruoyao Li, Zhiwei Feng, Qiantai Guo, Zijie Yang, Yuejin Wu, Hao Huang, Wenxuan Wei, Jiaqi Si, Dan Yao, Xiuqi Bu, Jia Huang, Haiwen Wang, Manning Fu, Tianfan Tang, Shixiang Fei, Ben Zhou, Dongzhan Ling, Fenghua Lu, Yan Sun, Siqi Li, Chenhui Zheng, Guanjie Lv, Jiancheng Zhang, Wenlong Bai, Lei Artificial Intelligence Computation and Language Scientific discoveries increasingly rely on complex multimodal reasoning based on information-intensive scientific data and domain-specific expertise. Empowered by expert-level scientific benchmarks, scientific Multimodal Large Language Models (MLLMs) hold the potential to significantly enhance this discovery process in realistic workflows. However, current scientific benchmarks mostly focus on evaluating the knowledge understanding capabilities of MLLMs, leading to an inadequate assessment of their perception and reasoning abilities. To address this gap, we present the Scientists' First Exam (SFE) benchmark, designed to evaluate the scientific cognitive capacities of MLLMs through three interconnected levels: scientific signal perception, scientific attribute understanding, scientific comparative reasoning. Specifically, SFE comprises 830 expert-verified VQA pairs across three question types, spanning 66 multimodal tasks across five high-value disciplines. Extensive experiments reveal that current state-of-the-art GPT-o3 and InternVL-3 achieve only 34.08% and 26.52% on SFE, highlighting significant room for MLLMs to improve in scientific realms. We hope the insights obtained in SFE will facilitate further developments in AI-enhanced scientific discoveries. |
| title | Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoning |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2506.10521 |