CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916327738834944 |
|---|---|
| author | Li, Zhong-Zhi Zhang, Ming-Liang Yin, Fei Ji, Zhi-Long Bai, Jin-Feng Pan, Zhen-Ru Zeng, Fan-Hu Xu, Jian Zhang, Jia-Xin Liu, Cheng-Lin |
| author_facet | Li, Zhong-Zhi Zhang, Ming-Liang Yin, Fei Ji, Zhi-Long Bai, Jin-Feng Pan, Zhen-Ru Zeng, Fan-Hu Xu, Jian Zhang, Jia-Xin Liu, Cheng-Lin |
| contents | Due to the rapid advancements in multimodal large language models, evaluating their multimodal mathematical capabilities continues to receive wide attention. Despite the datasets like MathVista proposed benchmarks for assessing mathematical capabilities in multimodal scenarios, there is still a lack of corresponding evaluation tools and datasets for fine-grained assessment in the context of K12 education in Chinese language. To systematically evaluate the capability of multimodal large models in solving Chinese multimodal mathematical problems, we propose a Chinese Multi-modal Math Skill Evaluation Benchmark, named CMMaTH, contraining 23k multimodal K12 math related questions, forming the largest Chinese multimodal mathematical problem benchmark to date. CMMaTH questions from elementary to high school levels, provide increased diversity in problem types, solution objectives, visual elements, detailed knowledge points, and standard solution annotations. We have constructed an open-source tool GradeGPT integrated with the CMMaTH dataset, facilitating stable, rapid, and cost-free model evaluation. Our data and code are available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12023 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models Li, Zhong-Zhi Zhang, Ming-Liang Yin, Fei Ji, Zhi-Long Bai, Jin-Feng Pan, Zhen-Ru Zeng, Fan-Hu Xu, Jian Zhang, Jia-Xin Liu, Cheng-Lin Computation and Language Artificial Intelligence Due to the rapid advancements in multimodal large language models, evaluating their multimodal mathematical capabilities continues to receive wide attention. Despite the datasets like MathVista proposed benchmarks for assessing mathematical capabilities in multimodal scenarios, there is still a lack of corresponding evaluation tools and datasets for fine-grained assessment in the context of K12 education in Chinese language. To systematically evaluate the capability of multimodal large models in solving Chinese multimodal mathematical problems, we propose a Chinese Multi-modal Math Skill Evaluation Benchmark, named CMMaTH, contraining 23k multimodal K12 math related questions, forming the largest Chinese multimodal mathematical problem benchmark to date. CMMaTH questions from elementary to high school levels, provide increased diversity in problem types, solution objectives, visual elements, detailed knowledge points, and standard solution annotations. We have constructed an open-source tool GradeGPT integrated with the CMMaTH dataset, facilitating stable, rapid, and cost-free model evaluation. Our data and code are available. |
| title | CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.12023 |