FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levels
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912949986131968 |
|---|---|
| author | Jiang, Jiedong He, Wanyi Wang, Yuefeng Gao, Guoxiong Hu, Yongle Wang, Jingting Guan, Nailin Wu, Peihao Dai, Chunbo Xiao, Liang Dong, Bin |
| author_facet | Jiang, Jiedong He, Wanyi Wang, Yuefeng Gao, Guoxiong Hu, Yongle Wang, Jingting Guan, Nailin Wu, Peihao Dai, Chunbo Xiao, Liang Dong, Bin |
| contents | Recent advances in large language models (LLMs) have demonstrated impressive capabilities in formal theorem proving, particularly on contest-based mathematical benchmarks like the IMO. However, these contests do not reflect the depth, breadth, and abstraction of modern mathematical research. To bridge this gap, we introduce FATE (Formal Algebra Theorem Evaluation), a new benchmark series in formal algebra designed to chart a course toward advanced mathematical reasoning. We present two new components, FATE-H and FATE-X, each with 100 problems in abstract and commutative algebra. The FATE series spans a difficulty spectrum from undergraduate exercises to problems exceeding PhD qualifying exams. Notably, FATE-X is the first formal benchmark to surpass both PhD-level exam difficulty and the coverage of the Mathlib library. Our evaluations of state-of-the-art LLM provers on this new benchmark reveal a stark performance gap compared to contest math: the best model achieves only 3% (pass@64) accuracy on FATE-H and 0% on FATE-X. Our two-stage evaluation reveals that models' natural-language reasoning is notably more accurate than their ability to formalize this reasoning. We systematically classify the common errors that arise during this formalization process. Furthermore, a comparative study shows that a specialized prover can exhibit less effective reflection than general-purpose models, reducing its accuracy at the natural-language stage. We believe FATE provides a robust and challenging benchmark that establishes essential checkpoints on the path toward research-level formal mathematical reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_02872 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levels Jiang, Jiedong He, Wanyi Wang, Yuefeng Gao, Guoxiong Hu, Yongle Wang, Jingting Guan, Nailin Wu, Peihao Dai, Chunbo Xiao, Liang Dong, Bin Machine Learning Artificial Intelligence Formal Languages and Automata Theory Logic in Computer Science Recent advances in large language models (LLMs) have demonstrated impressive capabilities in formal theorem proving, particularly on contest-based mathematical benchmarks like the IMO. However, these contests do not reflect the depth, breadth, and abstraction of modern mathematical research. To bridge this gap, we introduce FATE (Formal Algebra Theorem Evaluation), a new benchmark series in formal algebra designed to chart a course toward advanced mathematical reasoning. We present two new components, FATE-H and FATE-X, each with 100 problems in abstract and commutative algebra. The FATE series spans a difficulty spectrum from undergraduate exercises to problems exceeding PhD qualifying exams. Notably, FATE-X is the first formal benchmark to surpass both PhD-level exam difficulty and the coverage of the Mathlib library. Our evaluations of state-of-the-art LLM provers on this new benchmark reveal a stark performance gap compared to contest math: the best model achieves only 3% (pass@64) accuracy on FATE-H and 0% on FATE-X. Our two-stage evaluation reveals that models' natural-language reasoning is notably more accurate than their ability to formalize this reasoning. We systematically classify the common errors that arise during this formalization process. Furthermore, a comparative study shows that a specialized prover can exhibit less effective reflection than general-purpose models, reducing its accuracy at the natural-language stage. We believe FATE provides a robust and challenging benchmark that establishes essential checkpoints on the path toward research-level formal mathematical reasoning. |
| title | FATE: A Formal Benchmark Series for Frontier Algebra of Multiple Difficulty Levels |
| topic | Machine Learning Artificial Intelligence Formal Languages and Automata Theory Logic in Computer Science |
| url | https://arxiv.org/abs/2511.02872 |