Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910015509495808 |
|---|---|
| author | Qin, Yiwei Huang, Zhen Mi, Tiantian Si, Weiye Zhou, Chenyang Guo, Qipeng Feng, Siyuan Liu, Pengfei |
| author_facet | Qin, Yiwei Huang, Zhen Mi, Tiantian Si, Weiye Zhou, Chenyang Guo, Qipeng Feng, Siyuan Liu, Pengfei |
| contents | Data quality determines foundation model performance, yet systematic processing frameworks are lacking. We introduce Data Darwinism, a ten-level taxonomy (L0-L9) that conceptualizes data-model co-evolution: advanced models produce superior data for next-generation systems. We validate this on scientific literature by constructing Darwin-Science, a 900B-token corpus (L0-L5). We identify a learnability gap in raw scientific text, which we bridge via L4 (Generative Refinement) and L5 (Cognitive Completion) using frontier LLMs to explicate reasoning and terminology.
To ensure rigorous attribution, we pre-trained daVinci-origin-3B/7B models from scratch, excluding scientific content to create contamination-free baselines. After 600B tokens of continued pre-training, Darwin-Science outperforms baselines by +2.12 (3B) and +2.95 (7B) points across 20+ benchmarks, rising to +5.60 and +8.40 points on domain-aligned tasks. Systematic progression to L5 yields a +1.36 total gain, confirming that higher-level processing unlocks latent data value. We release the Darwin-Science corpus and daVinci-origin models to enable principled, co-evolutionary development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07824 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training Qin, Yiwei Huang, Zhen Mi, Tiantian Si, Weiye Zhou, Chenyang Guo, Qipeng Feng, Siyuan Liu, Pengfei Artificial Intelligence Computation and Language Data quality determines foundation model performance, yet systematic processing frameworks are lacking. We introduce Data Darwinism, a ten-level taxonomy (L0-L9) that conceptualizes data-model co-evolution: advanced models produce superior data for next-generation systems. We validate this on scientific literature by constructing Darwin-Science, a 900B-token corpus (L0-L5). We identify a learnability gap in raw scientific text, which we bridge via L4 (Generative Refinement) and L5 (Cognitive Completion) using frontier LLMs to explicate reasoning and terminology. To ensure rigorous attribution, we pre-trained daVinci-origin-3B/7B models from scratch, excluding scientific content to create contamination-free baselines. After 600B tokens of continued pre-training, Darwin-Science outperforms baselines by +2.12 (3B) and +2.95 (7B) points across 20+ benchmarks, rising to +5.60 and +8.40 points on domain-aligned tasks. Systematic progression to L5 yields a +1.36 total gain, confirming that higher-level processing unlocks latent data value. We release the Darwin-Science corpus and daVinci-origin models to enable principled, co-evolutionary development. |
| title | Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2602.07824 |