Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.18978 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918034170445824 |
|---|---|
| author | Perez, Miguel Angel Peñaloza Orozco, Bruno Lopez Soto, Jesus Tadeo Cruz Hernandez, Michelle Bruno Gonzalez, Miguel Angel Alvarado Malagon, Sandra |
| author_facet | Perez, Miguel Angel Peñaloza Orozco, Bruno Lopez Soto, Jesus Tadeo Cruz Hernandez, Michelle Bruno Gonzalez, Miguel Angel Alvarado Malagon, Sandra |
| contents | Existing mathematical reasoning benchmarks are predominantly English only or translation-based, which can introduce semantic drift and mask languagespecific reasoning errors. To address this, we present AI4Math, a benchmark of 105 original university level math problems natively authored in Spanish. The dataset spans seven advanced domains (Algebra, Calculus, Geometry, Probability, Number Theory, Combinatorics, and Logic), and each problem is accompanied by a step by step human solution. We evaluate six large language models GPT 4o, GPT 4o mini, o3 mini, LLaMA 3.3 70B, DeepSeek R1 685B, and DeepSeek V3 685B under four configurations: zero shot and chain of thought, each in Spanish and English. The top models (o3 mini, DeepSeek R1 685B, DeepSeek V3 685B) achieve over 70% accuracy, whereas LLaMA 3.3 70B and GPT-4o mini remain below 40%. Most models show no significant performance drop between languages, with GPT 4o even performing better on Spanish problems in the zero shot setting. Geometry, Combinatorics, and Probability questions remain persistently challenging for all models. These results highlight the need for native-language benchmarks and domain-specific evaluations to reveal reasoning failures not captured by standard metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18978 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AI4Math: A Native Spanish Benchmark for University-Level Mathematical Reasoning in Large Language Models Perez, Miguel Angel Peñaloza Orozco, Bruno Lopez Soto, Jesus Tadeo Cruz Hernandez, Michelle Bruno Gonzalez, Miguel Angel Alvarado Malagon, Sandra Computation and Language 68 I.2 Existing mathematical reasoning benchmarks are predominantly English only or translation-based, which can introduce semantic drift and mask languagespecific reasoning errors. To address this, we present AI4Math, a benchmark of 105 original university level math problems natively authored in Spanish. The dataset spans seven advanced domains (Algebra, Calculus, Geometry, Probability, Number Theory, Combinatorics, and Logic), and each problem is accompanied by a step by step human solution. We evaluate six large language models GPT 4o, GPT 4o mini, o3 mini, LLaMA 3.3 70B, DeepSeek R1 685B, and DeepSeek V3 685B under four configurations: zero shot and chain of thought, each in Spanish and English. The top models (o3 mini, DeepSeek R1 685B, DeepSeek V3 685B) achieve over 70% accuracy, whereas LLaMA 3.3 70B and GPT-4o mini remain below 40%. Most models show no significant performance drop between languages, with GPT 4o even performing better on Spanish problems in the zero shot setting. Geometry, Combinatorics, and Probability questions remain persistently challenging for all models. These results highlight the need for native-language benchmarks and domain-specific evaluations to reveal reasoning failures not captured by standard metrics. |
| title | AI4Math: A Native Spanish Benchmark for University-Level Mathematical Reasoning in Large Language Models |
| topic | Computation and Language 68 I.2 |
| url | https://arxiv.org/abs/2505.18978 |