Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.12575 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912658684379136 |
|---|---|
| author | Zhang, Jie Petrui, Cezara Nikolić, Kristina Tramèr, Florian |
| author_facet | Zhang, Jie Petrui, Cezara Nikolić, Kristina Tramèr, Florian |
| contents | Existing benchmarks for evaluating mathematical reasoning in large language models (LLMs) rely primarily on competition problems, formal proofs, or artificially challenging questions -- failing to capture the nature of mathematics encountered in actual research environments. We introduce RealMath, a novel benchmark derived directly from research papers and mathematical forums that assesses LLMs' abilities on authentic mathematical tasks. Our approach addresses three critical challenges: sourcing diverse research-level content, enabling reliable automated evaluation through verifiable statements, and designing a continually refreshable dataset to mitigate contamination risks. Experimental results across multiple LLMs reveal surprising capabilities in handling research mathematics compared to competition problems, suggesting current models may already serve as valuable assistants for working mathematicians despite limitations on highly challenging problems. The code and dataset for RealMath are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12575 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics Zhang, Jie Petrui, Cezara Nikolić, Kristina Tramèr, Florian Artificial Intelligence Existing benchmarks for evaluating mathematical reasoning in large language models (LLMs) rely primarily on competition problems, formal proofs, or artificially challenging questions -- failing to capture the nature of mathematics encountered in actual research environments. We introduce RealMath, a novel benchmark derived directly from research papers and mathematical forums that assesses LLMs' abilities on authentic mathematical tasks. Our approach addresses three critical challenges: sourcing diverse research-level content, enabling reliable automated evaluation through verifiable statements, and designing a continually refreshable dataset to mitigate contamination risks. Experimental results across multiple LLMs reveal surprising capabilities in handling research mathematics compared to competition problems, suggesting current models may already serve as valuable assistants for working mathematicians despite limitations on highly challenging problems. The code and dataset for RealMath are publicly available. |
| title | RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2505.12575 |