Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2509.24827 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918316736512000 |
|---|---|
| author | Bieganowski, Bartosz Strzelecki, Daniel Skiba, Robert Topolewski, Mateusz |
| author_facet | Bieganowski, Bartosz Strzelecki, Daniel Skiba, Robert Topolewski, Mateusz |
| contents | In this paper we summarize the results of the Putnam-like benchmark published by Google DeepMind. This dataset consists of 96 original problems in the spirit of the Putnam Competition and 576 solutions generated by LLMs. We analyze the performance of models on this set of problems to verify their ability to solve problems from mathematical contests. We find that top models, particularly Gemini 2.5 Pro, achieve high scores, demonstrating strong mathematical reasoning capabilities, although their performance was lower on problems from the 2024 Putnam competition. The analysis highlights distinct behavioral patterns among models, including bimodal scoring distributions and challenges in providing fully rigorous justifications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24827 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Putnam-like dataset summary: LLMs as mathematical competition contestants Bieganowski, Bartosz Strzelecki, Daniel Skiba, Robert Topolewski, Mateusz Machine Learning Artificial Intelligence In this paper we summarize the results of the Putnam-like benchmark published by Google DeepMind. This dataset consists of 96 original problems in the spirit of the Putnam Competition and 576 solutions generated by LLMs. We analyze the performance of models on this set of problems to verify their ability to solve problems from mathematical contests. We find that top models, particularly Gemini 2.5 Pro, achieve high scores, demonstrating strong mathematical reasoning capabilities, although their performance was lower on problems from the 2024 Putnam competition. The analysis highlights distinct behavioral patterns among models, including bimodal scoring distributions and challenges in providing fully rigorous justifications. |
| title | Putnam-like dataset summary: LLMs as mathematical competition contestants |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.24827 |