CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915004212576256 |
|---|---|
| author | Zhang, Ge Du, Xinrun Chen, Bei Liang, Yiming Luo, Tongxu Zheng, Tianyu Zhu, Kang Cheng, Yuyang Xu, Chunpu Guo, Shuyue Zhang, Haoran Qu, Xingwei Wang, Junjie Yuan, Ruibin Li, Yizhi Wang, Zekun Liu, Yudong Tsai, Yu-Hsuan Zhang, Fengji Lin, Chenghua Huang, Wenhao Fu, Jie |
| author_facet | Zhang, Ge Du, Xinrun Chen, Bei Liang, Yiming Luo, Tongxu Zheng, Tianyu Zhu, Kang Cheng, Yuyang Xu, Chunpu Guo, Shuyue Zhang, Haoran Qu, Xingwei Wang, Junjie Yuan, Ruibin Li, Yizhi Wang, Zekun Liu, Yudong Tsai, Yu-Hsuan Zhang, Fengji Lin, Chenghua Huang, Wenhao Fu, Jie |
| contents | As the capabilities of large multimodal models (LMMs) continue to advance, evaluating the performance of LMMs emerges as an increasing need. Additionally, there is an even larger gap in evaluating the advanced knowledge and reasoning abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to evaluate LMMs on tasks demanding college-level subject knowledge and deliberate reasoning in a Chinese context. CMMMU is inspired by and strictly follows the annotation and analysis pattern of MMMU. CMMMU includes 12k manually collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering, like its companion, MMMU. These questions span 30 subjects and comprise 39 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. CMMMU focuses on complex perception and reasoning with domain-specific knowledge in the Chinese context. We evaluate 11 open-source LLMs and one proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%, indicating a large space for improvement. CMMMU will boost the community to build the next-generation LMMs towards expert artificial intelligence and promote the democratization of LMMs by providing diverse language contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_11944 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark Zhang, Ge Du, Xinrun Chen, Bei Liang, Yiming Luo, Tongxu Zheng, Tianyu Zhu, Kang Cheng, Yuyang Xu, Chunpu Guo, Shuyue Zhang, Haoran Qu, Xingwei Wang, Junjie Yuan, Ruibin Li, Yizhi Wang, Zekun Liu, Yudong Tsai, Yu-Hsuan Zhang, Fengji Lin, Chenghua Huang, Wenhao Fu, Jie Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition As the capabilities of large multimodal models (LMMs) continue to advance, evaluating the performance of LMMs emerges as an increasing need. Additionally, there is an even larger gap in evaluating the advanced knowledge and reasoning abilities of LMMs in non-English contexts such as Chinese. We introduce CMMMU, a new Chinese Massive Multi-discipline Multimodal Understanding benchmark designed to evaluate LMMs on tasks demanding college-level subject knowledge and deliberate reasoning in a Chinese context. CMMMU is inspired by and strictly follows the annotation and analysis pattern of MMMU. CMMMU includes 12k manually collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering, like its companion, MMMU. These questions span 30 subjects and comprise 39 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. CMMMU focuses on complex perception and reasoning with domain-specific knowledge in the Chinese context. We evaluate 11 open-source LLMs and one proprietary GPT-4V(ision). Even GPT-4V only achieves accuracies of 42%, indicating a large space for improvement. CMMMU will boost the community to build the next-generation LMMs towards expert artificial intelligence and promote the democratization of LMMs by providing diverse language contexts. |
| title | CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.11944 |