LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915823634874368 |
|---|---|
| author | Peyronnet, Antoine Gloeckle, Fabian Hayat, Amaury |
| author_facet | Peyronnet, Antoine Gloeckle, Fabian Hayat, Amaury |
| contents | We present a new approach for benchmarking Large Language Model (LLM) capabilities on research-level mathematics. Existing benchmarks largely rely on static, hand-curated sets of contest or textbook-style problems as proxies for mathematical research. Instead, we establish an updatable benchmark evaluating models directly on the latest research results in mathematics. This consists of an automatic pipeline that extracts lemmas from arXiv and rewrites them into self-contained statements by making all assumptions and required definitions explicit. It results in a benchmark that can be updated regularly with new problems taken directly from human mathematical research, while previous instances can be used for training without compromising future evaluations. We benchmark current state-of-the-art LLMs, which obtain around 10-15$\%$ accuracy in theorem proving (pass@1) depending on the model, showing that there is currently a large margin of progression for LLMs to reach human-level proving capabilities in a research context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_24173 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics Peyronnet, Antoine Gloeckle, Fabian Hayat, Amaury Artificial Intelligence 68V15, 68T05, 68T50 I.2.3; I.2.7; H.3.3 We present a new approach for benchmarking Large Language Model (LLM) capabilities on research-level mathematics. Existing benchmarks largely rely on static, hand-curated sets of contest or textbook-style problems as proxies for mathematical research. Instead, we establish an updatable benchmark evaluating models directly on the latest research results in mathematics. This consists of an automatic pipeline that extracts lemmas from arXiv and rewrites them into self-contained statements by making all assumptions and required definitions explicit. It results in a benchmark that can be updated regularly with new problems taken directly from human mathematical research, while previous instances can be used for training without compromising future evaluations. We benchmark current state-of-the-art LLMs, which obtain around 10-15$\%$ accuracy in theorem proving (pass@1) depending on the model, showing that there is currently a large margin of progression for LLMs to reach human-level proving capabilities in a research context. |
| title | LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics |
| topic | Artificial Intelligence 68V15, 68T05, 68T50 I.2.3; I.2.7; H.3.3 |
| url | https://arxiv.org/abs/2602.24173 |