R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910927493791744 |
|---|---|
| author | Guo, Meng-Hao Xu, Jiajun Zhang, Yi Song, Jiaxi Peng, Haoyang Deng, Yi-Xuan Dong, Xinzhi Nakayama, Kiyohiro Geng, Zhengyang Wang, Chen Ni, Bolin Yang, Guo-Wei Rao, Yongming Peng, Houwen Hu, Han Wetzstein, Gordon Hu, Shi-min |
| author_facet | Guo, Meng-Hao Xu, Jiajun Zhang, Yi Song, Jiaxi Peng, Haoyang Deng, Yi-Xuan Dong, Xinzhi Nakayama, Kiyohiro Geng, Zhengyang Wang, Chen Ni, Bolin Yang, Guo-Wei Rao, Yongming Peng, Houwen Hu, Han Wetzstein, Gordon Hu, Shi-min |
| contents | Reasoning stands as a cornerstone of intelligence, enabling the synthesis of existing knowledge to solve complex problems. Despite remarkable progress, existing reasoning benchmarks often fail to rigorously evaluate the nuanced reasoning capabilities required for complex, real-world problemsolving, particularly in multi-disciplinary and multimodal contexts. In this paper, we introduce a graduate-level, multi-disciplinary, EnglishChinese benchmark, dubbed as Reasoning Bench (R-Bench), for assessing the reasoning capability of both language and multimodal models. RBench spans 1,094 questions across 108 subjects for language model evaluation and 665 questions across 83 subjects for multimodal model testing in both English and Chinese. These questions are meticulously curated to ensure rigorous difficulty calibration, subject balance, and crosslinguistic alignment, enabling the assessment to be an Olympiad-level multi-disciplinary benchmark. We evaluate widely used models, including OpenAI o1, GPT-4o, DeepSeek-R1, etc. Experimental results indicate that advanced models perform poorly on complex reasoning, especially multimodal reasoning. Even the top-performing model OpenAI o1 achieves only 53.2% accuracy on our multimodal evaluation. Data and code are made publicly available at here. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_02018 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation Guo, Meng-Hao Xu, Jiajun Zhang, Yi Song, Jiaxi Peng, Haoyang Deng, Yi-Xuan Dong, Xinzhi Nakayama, Kiyohiro Geng, Zhengyang Wang, Chen Ni, Bolin Yang, Guo-Wei Rao, Yongming Peng, Houwen Hu, Han Wetzstein, Gordon Hu, Shi-min Computer Vision and Pattern Recognition Reasoning stands as a cornerstone of intelligence, enabling the synthesis of existing knowledge to solve complex problems. Despite remarkable progress, existing reasoning benchmarks often fail to rigorously evaluate the nuanced reasoning capabilities required for complex, real-world problemsolving, particularly in multi-disciplinary and multimodal contexts. In this paper, we introduce a graduate-level, multi-disciplinary, EnglishChinese benchmark, dubbed as Reasoning Bench (R-Bench), for assessing the reasoning capability of both language and multimodal models. RBench spans 1,094 questions across 108 subjects for language model evaluation and 665 questions across 83 subjects for multimodal model testing in both English and Chinese. These questions are meticulously curated to ensure rigorous difficulty calibration, subject balance, and crosslinguistic alignment, enabling the assessment to be an Olympiad-level multi-disciplinary benchmark. We evaluate widely used models, including OpenAI o1, GPT-4o, DeepSeek-R1, etc. Experimental results indicate that advanced models perform poorly on complex reasoning, especially multimodal reasoning. Even the top-performing model OpenAI o1 achieves only 53.2% accuracy on our multimodal evaluation. Data and code are made publicly available at here. |
| title | R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.02018 |