Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2504.01995 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912320240746496 |
|---|---|
| author | Mahdavi, Hamed Hashemi, Alireza Daliri, Majid Mohammadipour, Pegah Farhadi, Alireza Malek, Samira Yazdanifard, Yekta Khasahmadi, Amir Honavar, Vasant |
| author_facet | Mahdavi, Hamed Hashemi, Alireza Daliri, Majid Mohammadipour, Pegah Farhadi, Alireza Malek, Samira Yazdanifard, Yekta Khasahmadi, Amir Honavar, Vasant |
| contents | Recent advances in large language models (LLMs) have shown impressive progress in mathematical reasoning tasks. However, current evaluation benchmarks predominantly focus on the accuracy of final answers, often overlooking the crucial logical rigor for mathematical problem solving. The claim that state-of-the-art LLMs can solve Math Olympiad-level problems requires closer examination. To explore this, we conducted both qualitative and quantitative human evaluations of proofs generated by LLMs, and developed a schema for automatically assessing their reasoning capabilities. Our study reveals that current LLMs fall significantly short of solving challenging Olympiad-level problems and frequently fail to distinguish correct mathematical reasoning from clearly flawed solutions. Our analyses demonstrate that the occasional correct final answers provided by LLMs often result from pattern recognition or heuristic shortcuts rather than genuine mathematical reasoning. These findings underscore the substantial gap between LLM performance and human expertise in advanced mathematical reasoning and highlight the importance of developing benchmarks that prioritize the soundness of the reasoning used to arrive at an answer rather than the mere correctness of the final answers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_01995 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics Mahdavi, Hamed Hashemi, Alireza Daliri, Majid Mohammadipour, Pegah Farhadi, Alireza Malek, Samira Yazdanifard, Yekta Khasahmadi, Amir Honavar, Vasant Artificial Intelligence Machine Learning Recent advances in large language models (LLMs) have shown impressive progress in mathematical reasoning tasks. However, current evaluation benchmarks predominantly focus on the accuracy of final answers, often overlooking the crucial logical rigor for mathematical problem solving. The claim that state-of-the-art LLMs can solve Math Olympiad-level problems requires closer examination. To explore this, we conducted both qualitative and quantitative human evaluations of proofs generated by LLMs, and developed a schema for automatically assessing their reasoning capabilities. Our study reveals that current LLMs fall significantly short of solving challenging Olympiad-level problems and frequently fail to distinguish correct mathematical reasoning from clearly flawed solutions. Our analyses demonstrate that the occasional correct final answers provided by LLMs often result from pattern recognition or heuristic shortcuts rather than genuine mathematical reasoning. These findings underscore the substantial gap between LLM performance and human expertise in advanced mathematical reasoning and highlight the importance of developing benchmarks that prioritize the soundness of the reasoning used to arrive at an answer rather than the mere correctness of the final answers. |
| title | Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2504.01995 |