Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.08744 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916735487049728 |
|---|---|
| author | Chen, Xiaoyang Dai, Xinan Du, Yu Feng, Qian Guo, Naixu Gu, Tingshuo Gao, Yuting Gao, Yingyi Han, Xudong Jiang, Xiang Jin, Yilin Lin, Hongyi Lin, Shisheng Li, Xiangnan Li, Yuante Li, Yixing Lai, Zhentao Ma, Zilu Peng, Yingrong Qian, Jiacheng Sun, Hao-Yu Sun, Jianbo Wang, Zirui Wu, Siwei Wang, Zian Xu, Bin Xu, Jianghao Yu, Yiyang Yang, Zichuan Zha, Hongji Zhang, Ruichong |
| author_facet | Chen, Xiaoyang Dai, Xinan Du, Yu Feng, Qian Guo, Naixu Gu, Tingshuo Gao, Yuting Gao, Yingyi Han, Xudong Jiang, Xiang Jin, Yilin Lin, Hongyi Lin, Shisheng Li, Xiangnan Li, Yuante Li, Yixing Lai, Zhentao Ma, Zilu Peng, Yingrong Qian, Jiacheng Sun, Hao-Yu Sun, Jianbo Wang, Zirui Wu, Siwei Wang, Zian Xu, Bin Xu, Jianghao Yu, Yiyang Yang, Zichuan Zha, Hongji Zhang, Ruichong |
| contents | To advance the mathematical proficiency of large language models (LLMs), the DeepMath team has launched an open-source initiative aimed at developing an open mathematical LLM and systematically evaluating its mathematical creativity. This paper represents the initial contribution of this initiative. While recent developments in mathematical LLMs have predominantly emphasized reasoning skills, as evidenced by benchmarks on elementary to undergraduate-level mathematical tasks, the creative capabilities of these models have received comparatively little attention, and evaluation datasets remain scarce. To address this gap, we propose an evaluation criteria for mathematical creativity and introduce DeepMath-Creative, a novel, high-quality benchmark comprising constructive problems across algebra, geometry, analysis, and other domains. We conduct a systematic evaluation of mainstream LLMs' creative problem-solving abilities using this dataset. Experimental results show that even under lenient scoring criteria -- emphasizing core solution components and disregarding minor inaccuracies, such as small logical gaps, incomplete justifications, or redundant explanations -- the best-performing model, O3 Mini, achieves merely 70% accuracy, primarily on basic undergraduate-level constructive tasks. Performance declines sharply on more complex problems, with models failing to provide substantive strategies for open problems. These findings suggest that, although current LLMs display a degree of constructive proficiency on familiar and lower-difficulty problems, such performance is likely attributable to the recombination of memorized patterns rather than authentic creative insight or novel synthesis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_08744 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DeepMath-Creative: A Benchmark for Evaluating Mathematical Creativity of Large Language Models Chen, Xiaoyang Dai, Xinan Du, Yu Feng, Qian Guo, Naixu Gu, Tingshuo Gao, Yuting Gao, Yingyi Han, Xudong Jiang, Xiang Jin, Yilin Lin, Hongyi Lin, Shisheng Li, Xiangnan Li, Yuante Li, Yixing Lai, Zhentao Ma, Zilu Peng, Yingrong Qian, Jiacheng Sun, Hao-Yu Sun, Jianbo Wang, Zirui Wu, Siwei Wang, Zian Xu, Bin Xu, Jianghao Yu, Yiyang Yang, Zichuan Zha, Hongji Zhang, Ruichong Artificial Intelligence To advance the mathematical proficiency of large language models (LLMs), the DeepMath team has launched an open-source initiative aimed at developing an open mathematical LLM and systematically evaluating its mathematical creativity. This paper represents the initial contribution of this initiative. While recent developments in mathematical LLMs have predominantly emphasized reasoning skills, as evidenced by benchmarks on elementary to undergraduate-level mathematical tasks, the creative capabilities of these models have received comparatively little attention, and evaluation datasets remain scarce. To address this gap, we propose an evaluation criteria for mathematical creativity and introduce DeepMath-Creative, a novel, high-quality benchmark comprising constructive problems across algebra, geometry, analysis, and other domains. We conduct a systematic evaluation of mainstream LLMs' creative problem-solving abilities using this dataset. Experimental results show that even under lenient scoring criteria -- emphasizing core solution components and disregarding minor inaccuracies, such as small logical gaps, incomplete justifications, or redundant explanations -- the best-performing model, O3 Mini, achieves merely 70% accuracy, primarily on basic undergraduate-level constructive tasks. Performance declines sharply on more complex problems, with models failing to provide substantive strategies for open problems. These findings suggest that, although current LLMs display a degree of constructive proficiency on familiar and lower-difficulty problems, such performance is likely attributable to the recombination of memorized patterns rather than authentic creative insight or novel synthesis. |
| title | DeepMath-Creative: A Benchmark for Evaluating Mathematical Creativity of Large Language Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2505.08744 |