FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911334021464064 |
|---|---|
| author | Glazer, Elliot Erdil, Ege Besiroglu, Tamay Chicharro, Diego Chen, Evan Gunning, Alex Olsson, Caroline Falkman Denain, Jean-Stanislas Ho, Anson Santos, Emily de Oliveira Järviniemi, Olli Barnett, Matthew Sandler, Robert Vrzala, Matej Sevilla, Jaime Ren, Qiuyu Pratt, Elizabeth Levine, Lionel Barkley, Grant Stewart, Natalie Grechuk, Bogdan Grechuk, Tetiana Enugandla, Shreepranav Varma Wildon, Mark |
| author_facet | Glazer, Elliot Erdil, Ege Besiroglu, Tamay Chicharro, Diego Chen, Evan Gunning, Alex Olsson, Caroline Falkman Denain, Jean-Stanislas Ho, Anson Santos, Emily de Oliveira Järviniemi, Olli Barnett, Matthew Sandler, Robert Vrzala, Matej Sevilla, Jaime Ren, Qiuyu Pratt, Elizabeth Levine, Lionel Barkley, Grant Stewart, Natalie Grechuk, Bogdan Grechuk, Tetiana Enugandla, Shreepranav Varma Wildon, Mark |
| contents | We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_04872 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI Glazer, Elliot Erdil, Ege Besiroglu, Tamay Chicharro, Diego Chen, Evan Gunning, Alex Olsson, Caroline Falkman Denain, Jean-Stanislas Ho, Anson Santos, Emily de Oliveira Järviniemi, Olli Barnett, Matthew Sandler, Robert Vrzala, Matej Sevilla, Jaime Ren, Qiuyu Pratt, Elizabeth Levine, Lionel Barkley, Grant Stewart, Natalie Grechuk, Bogdan Grechuk, Tetiana Enugandla, Shreepranav Varma Wildon, Mark Artificial Intelligence We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress. |
| title | FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2411.04872 |