Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2410.09988 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910744501551104 |
|---|---|
| author | Fan, Jingxuan Martinson, Sarah Wang, Erik Y. Hausknecht, Kaylie Brenner, Jonah Liu, Danxian Peng, Nianli Wang, Corey Brenner, Michael P. |
| author_facet | Fan, Jingxuan Martinson, Sarah Wang, Erik Y. Hausknecht, Kaylie Brenner, Jonah Liu, Danxian Peng, Nianli Wang, Corey Brenner, Michael P. |
| contents | Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMath, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on HARDMath-mini, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like HARDMath to advance mathematical abilities of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09988 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics Fan, Jingxuan Martinson, Sarah Wang, Erik Y. Hausknecht, Kaylie Brenner, Jonah Liu, Danxian Peng, Nianli Wang, Corey Brenner, Michael P. Machine Learning Artificial Intelligence Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMath, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on HARDMath-mini, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like HARDMath to advance mathematical abilities of LLMs. |
| title | HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.09988 |