We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909236291698688 |
|---|---|
| author | Qiao, Runqi Tan, Qiuna Dong, Guanting Wu, Minhui Sun, Chong Song, Xiaoshuai GongQue, Zhuoma Lei, Shanglin Wei, Zhe Zhang, Miaoxuan Qiao, Runfeng Zhang, Yifan Zong, Xiao Xu, Yida Diao, Muxi Bao, Zhimin Li, Chen Zhang, Honggang |
| author_facet | Qiao, Runqi Tan, Qiuna Dong, Guanting Wu, Minhui Sun, Chong Song, Xiaoshuai GongQue, Zhuoma Lei, Shanglin Wei, Zhe Zhang, Miaoxuan Qiao, Runfeng Zhang, Yifan Zong, Xiao Xu, Yida Diao, Muxi Bao, Zhimin Li, Chen Zhang, Honggang |
| contents | Visual mathematical reasoning, as a fundamental visual reasoning ability, has received widespread attention from the Large Multimodal Models (LMMs) community. Existing benchmarks, such as MathVista and MathVerse, focus more on the result-oriented performance but neglect the underlying principles in knowledge acquisition and generalization. Inspired by human-like mathematical reasoning, we introduce WE-MATH, the first benchmark specifically designed to explore the problem-solving principles beyond end-to-end performance. We meticulously collect and categorize 6.5K visual math problems, spanning 67 hierarchical knowledge concepts and five layers of knowledge granularity. We decompose composite problems into sub-problems according to the required knowledge concepts and introduce a novel four-dimensional metric, namely Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), and Rote Memorization (RM), to hierarchically assess inherent issues in LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of existing LMMs in visual mathematical reasoning and reveal a negative correlation between solving steps and problem-specific performance. We confirm the IK issue of LMMs can be effectively improved via knowledge augmentation strategies. More notably, the primary challenge of GPT-4o has significantly transitioned from IK to IG, establishing it as the first LMM advancing towards the knowledge generalization stage. In contrast, other LMMs exhibit a marked inclination towards Rote Memorization - they correctly solve composite problems involving multiple knowledge concepts yet fail to answer sub-problems. We anticipate that WE-MATH will open new pathways for advancements in visual mathematical reasoning for LMMs. The WE-MATH data and evaluation code are available at https://github.com/We-Math/We-Math. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01284 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? Qiao, Runqi Tan, Qiuna Dong, Guanting Wu, Minhui Sun, Chong Song, Xiaoshuai GongQue, Zhuoma Lei, Shanglin Wei, Zhe Zhang, Miaoxuan Qiao, Runfeng Zhang, Yifan Zong, Xiao Xu, Yida Diao, Muxi Bao, Zhimin Li, Chen Zhang, Honggang Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Symbolic Computation Visual mathematical reasoning, as a fundamental visual reasoning ability, has received widespread attention from the Large Multimodal Models (LMMs) community. Existing benchmarks, such as MathVista and MathVerse, focus more on the result-oriented performance but neglect the underlying principles in knowledge acquisition and generalization. Inspired by human-like mathematical reasoning, we introduce WE-MATH, the first benchmark specifically designed to explore the problem-solving principles beyond end-to-end performance. We meticulously collect and categorize 6.5K visual math problems, spanning 67 hierarchical knowledge concepts and five layers of knowledge granularity. We decompose composite problems into sub-problems according to the required knowledge concepts and introduce a novel four-dimensional metric, namely Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), and Rote Memorization (RM), to hierarchically assess inherent issues in LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of existing LMMs in visual mathematical reasoning and reveal a negative correlation between solving steps and problem-specific performance. We confirm the IK issue of LMMs can be effectively improved via knowledge augmentation strategies. More notably, the primary challenge of GPT-4o has significantly transitioned from IK to IG, establishing it as the first LMM advancing towards the knowledge generalization stage. In contrast, other LMMs exhibit a marked inclination towards Rote Memorization - they correctly solve composite problems involving multiple knowledge concepts yet fail to answer sub-problems. We anticipate that WE-MATH will open new pathways for advancements in visual mathematical reasoning for LMMs. The WE-MATH data and evaluation code are available at https://github.com/We-Math/We-Math. |
| title | We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Symbolic Computation |
| url | https://arxiv.org/abs/2407.01284 |