More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909831494893568 |
|---|---|
| author | Zhao, Yike Guo, Simin Yang, Ziqing Han, Shifan Lin, Dahua Tan, Fei |
| author_facet | Zhao, Yike Guo, Simin Yang, Ziqing Han, Shifan Lin, Dahua Tan, Fei |
| contents | The reasoning capabilities of Large Language Models (LLMs) play a critical role in many downstream tasks, yet depend strongly on the quality of training data. Despite various proposed data construction methods, their practical utility in real-world pipelines remains underexplored. In this work, we conduct a comprehensive analysis of open-source datasets and data synthesis techniques for mathematical reasoning, evaluating them under a unified pipeline designed to mirror training and deployment scenarios. We further distill effective data selection strategies and identify practical methods suitable for industrial applications. Our findings highlight that structuring data in more interpretable formats, or distilling from stronger models often outweighs simply scaling up data volume. This study provides actionable guidance for integrating training data to enhance LLM capabilities, supporting both cost-effective data curation and scalable model enhancement. We hope this work will inspire further research on how to balance "more data" versus "better data" for real-world reasoning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07169 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning Zhao, Yike Guo, Simin Yang, Ziqing Han, Shifan Lin, Dahua Tan, Fei Computation and Language The reasoning capabilities of Large Language Models (LLMs) play a critical role in many downstream tasks, yet depend strongly on the quality of training data. Despite various proposed data construction methods, their practical utility in real-world pipelines remains underexplored. In this work, we conduct a comprehensive analysis of open-source datasets and data synthesis techniques for mathematical reasoning, evaluating them under a unified pipeline designed to mirror training and deployment scenarios. We further distill effective data selection strategies and identify practical methods suitable for industrial applications. Our findings highlight that structuring data in more interpretable formats, or distilling from stronger models often outweighs simply scaling up data volume. This study provides actionable guidance for integrating training data to enhance LLM capabilities, supporting both cost-effective data curation and scalable model enhancement. We hope this work will inspire further research on how to balance "more data" versus "better data" for real-world reasoning tasks. |
| title | More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.07169 |