CombiBench: Benchmarking LLM Capability for Combinatorial Mathematics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908351547310080 |
|---|---|
| author | Liu, Junqi Lin, Xiaohan Bayer, Jonas Dillies, Yael Jiang, Weijie Liang, Xiaodan Soletskyi, Roman Wang, Haiming Xie, Yunzhou Xiong, Beibei Yang, Zhengfeng Zhang, Jujian Zhi, Lihong Li, Jia Liu, Zhengying |
| author_facet | Liu, Junqi Lin, Xiaohan Bayer, Jonas Dillies, Yael Jiang, Weijie Liang, Xiaodan Soletskyi, Roman Wang, Haiming Xie, Yunzhou Xiong, Beibei Yang, Zhengfeng Zhang, Jujian Zhi, Lihong Li, Jia Liu, Zhengying |
| contents | Neurosymbolic approaches integrating large language models with formal reasoning have recently achieved human-level performance on mathematics competition problems in algebra, geometry and number theory. In comparison, combinatorics remains a challenging domain, characterized by a lack of appropriate benchmarks and theorem libraries. To address this gap, we introduce CombiBench, a comprehensive benchmark comprising 100 combinatorial problems, each formalized in Lean~4 and paired with its corresponding informal statement. The problem set covers a wide spectrum of difficulty levels, ranging from middle school to IMO and university level, and span over ten combinatorial topics. CombiBench is suitable for testing IMO solving capabilities since it includes all IMO combinatorial problems since 2000 (except IMO 2004 P3 as its statement contain an images). Furthermore, we provide a comprehensive and standardized evaluation framework, dubbed Fine-Eval (for $\textbf{F}$ill-in-the-blank $\textbf{in}$ L$\textbf{e}$an Evaluation), for formal mathematics. It accommodates not only proof-based problems but also, for the first time, the evaluation of fill-in-the-blank questions. Using Fine-Eval as the evaluation method and Kimina Lean Server as the backend, we benchmark several LLMs on CombiBench and observe that their capabilities for formally solving combinatorial problems remain limited. Among all models tested (none of which has been trained for this particular task), Kimina-Prover attains the best results, solving 7 problems (out of 100) under both ``with solution'' and ``without solution'' scenarios. We open source the benchmark dataset alongside with the code of the proposed evaluation method at https://github.com/MoonshotAI/CombiBench/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_03171 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CombiBench: Benchmarking LLM Capability for Combinatorial Mathematics Liu, Junqi Lin, Xiaohan Bayer, Jonas Dillies, Yael Jiang, Weijie Liang, Xiaodan Soletskyi, Roman Wang, Haiming Xie, Yunzhou Xiong, Beibei Yang, Zhengfeng Zhang, Jujian Zhi, Lihong Li, Jia Liu, Zhengying Artificial Intelligence Neurosymbolic approaches integrating large language models with formal reasoning have recently achieved human-level performance on mathematics competition problems in algebra, geometry and number theory. In comparison, combinatorics remains a challenging domain, characterized by a lack of appropriate benchmarks and theorem libraries. To address this gap, we introduce CombiBench, a comprehensive benchmark comprising 100 combinatorial problems, each formalized in Lean~4 and paired with its corresponding informal statement. The problem set covers a wide spectrum of difficulty levels, ranging from middle school to IMO and university level, and span over ten combinatorial topics. CombiBench is suitable for testing IMO solving capabilities since it includes all IMO combinatorial problems since 2000 (except IMO 2004 P3 as its statement contain an images). Furthermore, we provide a comprehensive and standardized evaluation framework, dubbed Fine-Eval (for $\textbf{F}$ill-in-the-blank $\textbf{in}$ L$\textbf{e}$an Evaluation), for formal mathematics. It accommodates not only proof-based problems but also, for the first time, the evaluation of fill-in-the-blank questions. Using Fine-Eval as the evaluation method and Kimina Lean Server as the backend, we benchmark several LLMs on CombiBench and observe that their capabilities for formally solving combinatorial problems remain limited. Among all models tested (none of which has been trained for this particular task), Kimina-Prover attains the best results, solving 7 problems (out of 100) under both ``with solution'' and ``without solution'' scenarios. We open source the benchmark dataset alongside with the code of the proposed evaluation method at https://github.com/MoonshotAI/CombiBench/. |
| title | CombiBench: Benchmarking LLM Capability for Combinatorial Mathematics |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2505.03171 |