Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915729087922176 |
|---|---|
| author | Krishnakumar, Arjun Sukthanker, Rhea Sanjay Mahadik, Hannan Javed Kadlecová, Gabriela Moroshan, Vladyslav Carstensen, Timur Hutter, Frank Klein, Aaron |
| author_facet | Krishnakumar, Arjun Sukthanker, Rhea Sanjay Mahadik, Hannan Javed Kadlecová, Gabriela Moroshan, Vladyslav Carstensen, Timur Hutter, Frank Klein, Aaron |
| contents | Small Language models (SLMs) offer an efficient and accessible alternative to Large Language Models (LLMs), delivering strong performance while using far fewer resources. We introduce a simple and effective framework for pretraining SLMs that brings together three complementary ideas. First, we identify structurally sparse sub-network initializations that consistently outperform randomly initialized models of similar size under the same compute budget. Second, we use evolutionary search to automatically discover high-quality sub-network initializations, providing better starting points for pretraining. Third, we apply knowledge distillation from larger teacher models to speed up training and improve generalization. Together, these components make SLM pretraining substantially more efficient: our best model, discovered using evolutionary search and initialized with LLM weights, matches the validation perplexity of a comparable Pythia SLM while requiring 5.16x and 1.26x fewer floating point operations for token budgets of 10B and 100B, respectively. We release all code publicly, offering a practical and reproducible path toward cost-efficient small language model development at scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07227 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation Krishnakumar, Arjun Sukthanker, Rhea Sanjay Mahadik, Hannan Javed Kadlecová, Gabriela Moroshan, Vladyslav Carstensen, Timur Hutter, Frank Klein, Aaron Computation and Language Artificial Intelligence Small Language models (SLMs) offer an efficient and accessible alternative to Large Language Models (LLMs), delivering strong performance while using far fewer resources. We introduce a simple and effective framework for pretraining SLMs that brings together three complementary ideas. First, we identify structurally sparse sub-network initializations that consistently outperform randomly initialized models of similar size under the same compute budget. Second, we use evolutionary search to automatically discover high-quality sub-network initializations, providing better starting points for pretraining. Third, we apply knowledge distillation from larger teacher models to speed up training and improve generalization. Together, these components make SLM pretraining substantially more efficient: our best model, discovered using evolutionary search and initialized with LLM weights, matches the validation perplexity of a comparable Pythia SLM while requiring 5.16x and 1.26x fewer floating point operations for token budgets of 10B and 100B, respectively. We release all code publicly, offering a practical and reproducible path toward cost-efficient small language model development at scale. |
| title | Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.07227 |