Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.13977 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918448678830080 |
|---|---|
| author | Niklaus, Joel Yamaguchi, Atsuki Štefánik, Michal Penedo, Guilherme Kydlíček, Hynek Bakouch, Elie Tunstall, Lewis Beeching, Edward Emanuel Frere, Thibaud Raffel, Colin von Werra, Leandro Wolf, Thomas |
| author_facet | Niklaus, Joel Yamaguchi, Atsuki Štefánik, Michal Penedo, Guilherme Kydlíček, Hynek Bakouch, Elie Tunstall, Lewis Beeching, Edward Emanuel Frere, Thibaud Raffel, Colin von Werra, Leandro Wolf, Thomas |
| contents | Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop \textbf{\textsc{FinePhrase}}, a 486-billion-token open dataset of rephrased web text. We show that \textsc{FinePhrase} outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_13977 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data Niklaus, Joel Yamaguchi, Atsuki Štefánik, Michal Penedo, Guilherme Kydlíček, Hynek Bakouch, Elie Tunstall, Lewis Beeching, Edward Emanuel Frere, Thibaud Raffel, Colin von Werra, Leandro Wolf, Thomas Computation and Language Artificial Intelligence Machine Learning Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop \textbf{\textsc{FinePhrase}}, a 486-billion-token open dataset of rephrased web text. We show that \textsc{FinePhrase} outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community. |
| title | How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.13977 |