Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910036128694272 |
|---|---|
| author | Fujii, Kazuki Tajima, Yukito Mizuki, Sakae Kawamura, Masaki Shimada, Hinari Shiotani, Taihei Saito, Koshiro Oi, Masanari Nakamura, Taishi Okamoto, Takumi Ishida, Shigeki Hattori, Kakeru Ma, Youmi Takamura, Hiroya Yokota, Rio Sakuma, Jun Okazaki, Naoaki |
| author_facet | Fujii, Kazuki Tajima, Yukito Mizuki, Sakae Kawamura, Masaki Shimada, Hinari Shiotani, Taihei Saito, Koshiro Oi, Masanari Nakamura, Taishi Okamoto, Takumi Ishida, Shigeki Hattori, Kakeru Ma, Youmi Takamura, Hiroya Yokota, Rio Sakuma, Jun Okazaki, Naoaki |
| contents | The performance of large language models (LLMs) in program synthesis and mathematical reasoning is fundamentally limited by the quality of their pre-training corpora. We introduce two openly licensed pre-training datasets, released under the Llama 3.3 Community License, that significantly enhance LLM performance by systematically rewriting public data. SwallowCode ($\approx$16.1 billion tokens) refines Python snippets from The-Stack-v2 through a novel four-stage pipeline: syntax validation, pylint-based style filtering, and a two-stage LLM rewriting process that enforces style conformity and transforms snippets into self-contained, algorithmically efficient examples. Unlike prior methods that rely on exclusionary filtering or limited transformations, our transform-and-retain approach refines low-quality code, maximizing data utility. SwallowMath ($\approx$2.3 billion tokens) enhances Finemath-4+ by removing boilerplate, restoring context, and reformatting solutions into concise, step-by-step explanations. Within a fixed 50 billion token training budget, continual pre-training of Llama-3.1-8B with SwallowCode boosts pass@1 by +17.0 on HumanEval and +16.1 on HumanEval+ compared to Stack-Edu, surpassing the baseline model's code generation capabilities. Similarly, substituting SwallowMath yields +12.4 accuracy on GSM8K and +7.6 on MATH. Ablation studies confirm that each pipeline stage contributes incrementally, with rewriting yielding the largest gains. By releasing datasets, prompts, checkpoints, and pipeline code, we ensure reproducibility and provide a transferable transform-and-retain methodology that can be adapted to other base models and LLM rewriting setups. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_02881 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rewriting Pre-Training Data Boosts LLM Performance in Math and Code Fujii, Kazuki Tajima, Yukito Mizuki, Sakae Kawamura, Masaki Shimada, Hinari Shiotani, Taihei Saito, Koshiro Oi, Masanari Nakamura, Taishi Okamoto, Takumi Ishida, Shigeki Hattori, Kakeru Ma, Youmi Takamura, Hiroya Yokota, Rio Sakuma, Jun Okazaki, Naoaki Machine Learning Artificial Intelligence The performance of large language models (LLMs) in program synthesis and mathematical reasoning is fundamentally limited by the quality of their pre-training corpora. We introduce two openly licensed pre-training datasets, released under the Llama 3.3 Community License, that significantly enhance LLM performance by systematically rewriting public data. SwallowCode ($\approx$16.1 billion tokens) refines Python snippets from The-Stack-v2 through a novel four-stage pipeline: syntax validation, pylint-based style filtering, and a two-stage LLM rewriting process that enforces style conformity and transforms snippets into self-contained, algorithmically efficient examples. Unlike prior methods that rely on exclusionary filtering or limited transformations, our transform-and-retain approach refines low-quality code, maximizing data utility. SwallowMath ($\approx$2.3 billion tokens) enhances Finemath-4+ by removing boilerplate, restoring context, and reformatting solutions into concise, step-by-step explanations. Within a fixed 50 billion token training budget, continual pre-training of Llama-3.1-8B with SwallowCode boosts pass@1 by +17.0 on HumanEval and +16.1 on HumanEval+ compared to Stack-Edu, surpassing the baseline model's code generation capabilities. Similarly, substituting SwallowMath yields +12.4 accuracy on GSM8K and +7.6 on MATH. Ablation studies confirm that each pipeline stage contributes incrementally, with rewriting yielding the largest gains. By releasing datasets, prompts, checkpoints, and pipeline code, we ensure reproducibility and provide a transferable transform-and-retain methodology that can be adapted to other base models and LLM rewriting setups. |
| title | Rewriting Pre-Training Data Boosts LLM Performance in Math and Code |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.02881 |