Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912570974142464 |
|---|---|
| author | Petrov, Ivo Dekoninck, Jasper Baltadzhiev, Lyuben Drencheva, Maria Minchev, Kristian Balunović, Mislav Jovanović, Nikola Vechev, Martin |
| author_facet | Petrov, Ivo Dekoninck, Jasper Baltadzhiev, Lyuben Drencheva, Maria Minchev, Kristian Balunović, Mislav Jovanović, Nikola Vechev, Martin |
| contents | Recent math benchmarks for large language models (LLMs) such as MathArena indicate that state-of-the-art reasoning models achieve impressive performance on mathematical competitions like AIME, with the leading model, Gemini-2.5-Pro, achieving scores comparable to top human competitors. However, these benchmarks evaluate models solely based on final numerical answers, neglecting rigorous reasoning and proof generation which are essential for real-world mathematical tasks. To address this, we introduce a comprehensive evaluation of full-solution reasoning for challenging mathematical problems. Using expert human annotators, we evaluated several state-of-the-art reasoning models on the six problems from the 2025 USAMO within hours of their release. Our results reveal that all tested models struggled significantly: only Gemini-2.5-Pro achieves a non-trivial score of 25%, while all other models achieve less than 5%. Through detailed analysis of reasoning traces, we identify the most common failure modes and find several unwanted artifacts arising from the optimization strategies employed during model training. Overall, our results suggest that current LLMs are inadequate for rigorous mathematical reasoning tasks, highlighting the need for substantial improvements in reasoning and proof generation capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_21934 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad Petrov, Ivo Dekoninck, Jasper Baltadzhiev, Lyuben Drencheva, Maria Minchev, Kristian Balunović, Mislav Jovanović, Nikola Vechev, Martin Computation and Language Recent math benchmarks for large language models (LLMs) such as MathArena indicate that state-of-the-art reasoning models achieve impressive performance on mathematical competitions like AIME, with the leading model, Gemini-2.5-Pro, achieving scores comparable to top human competitors. However, these benchmarks evaluate models solely based on final numerical answers, neglecting rigorous reasoning and proof generation which are essential for real-world mathematical tasks. To address this, we introduce a comprehensive evaluation of full-solution reasoning for challenging mathematical problems. Using expert human annotators, we evaluated several state-of-the-art reasoning models on the six problems from the 2025 USAMO within hours of their release. Our results reveal that all tested models struggled significantly: only Gemini-2.5-Pro achieves a non-trivial score of 25%, while all other models achieve less than 5%. Through detailed analysis of reasoning traces, we identify the most common failure modes and find several unwanted artifacts arising from the optimization strategies employed during model training. Overall, our results suggest that current LLMs are inadequate for rigorous mathematical reasoning tasks, highlighting the need for substantial improvements in reasoning and proof generation capabilities. |
| title | Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.21934 |