MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908389155536896 |
|---|---|
| author | Ye, Xinwu Li, Chengfan Chen, Siming Wei, Wei Tang, Xiangru |
| author_facet | Ye, Xinwu Li, Chengfan Chen, Siming Wei, Wei Tang, Xiangru |
| contents | Recent advances in large language models (LLMs) and vision-language models (LVLMs) have shown promise across many tasks, yet their scientific reasoning capabilities remain untested, particularly in multimodal settings. We present MMSciBench, a benchmark for evaluating mathematical and physical reasoning through text-only and text-image formats, with human-annotated difficulty levels, solutions with detailed explanations, and taxonomic mappings. Evaluation of state-of-the-art models reveals significant limitations, with even the best model achieving only \textbf{63.77\%} accuracy and particularly struggling with visual reasoning tasks. Our analysis exposes critical gaps in complex reasoning and visual-textual integration, establishing MMSciBench as a rigorous standard for measuring progress in multimodal scientific understanding. The code for MMSciBench is open-sourced at GitHub, and the dataset is available at Hugging Face. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01891 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems Ye, Xinwu Li, Chengfan Chen, Siming Wei, Wei Tang, Xiangru Machine Learning Computation and Language I.2.7 Recent advances in large language models (LLMs) and vision-language models (LVLMs) have shown promise across many tasks, yet their scientific reasoning capabilities remain untested, particularly in multimodal settings. We present MMSciBench, a benchmark for evaluating mathematical and physical reasoning through text-only and text-image formats, with human-annotated difficulty levels, solutions with detailed explanations, and taxonomic mappings. Evaluation of state-of-the-art models reveals significant limitations, with even the best model achieving only \textbf{63.77\%} accuracy and particularly struggling with visual reasoning tasks. Our analysis exposes critical gaps in complex reasoning and visual-textual integration, establishing MMSciBench as a rigorous standard for measuring progress in multimodal scientific understanding. The code for MMSciBench is open-sourced at GitHub, and the dataset is available at Hugging Face. |
| title | MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems |
| topic | Machine Learning Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2503.01891 |