Optimal Splitting of Language Models from Mixtures to Specialized Domains
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914409416228864 |
|---|---|
| author | Seto, Skyler Ablin, Pierre Filippova, Anastasiia Ye, Jiayuan Bethune, Louis Katharopoulos, Angelos Grangier, David |
| author_facet | Seto, Skyler Ablin, Pierre Filippova, Anastasiia Ye, Jiayuan Bethune, Louis Katharopoulos, Angelos Grangier, David |
| contents | Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard training recipe is a two-stage paradigm: pretraining first on the full corpus of data followed by specialization on a subset of high quality, specialized data from the full corpus. In the multi-domain setting, this involves continued pretraining of multiple models on each specialized domain, referred to as split model training. We propose a method for pretraining multiple models independently over a general pretraining corpus, and determining the optimal compute allocation between pretraining and continued pretraining using scaling laws. Our approach accurately predicts the loss of a model of size N with D pretraining and D' specialization tokens, and extrapolates to larger model sizes and number of tokens. Applied to language model training, our approach improves performance consistently across common sense knowledge and reasoning benchmarks across different model sizes and compute budgets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_19149 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Optimal Splitting of Language Models from Mixtures to Specialized Domains Seto, Skyler Ablin, Pierre Filippova, Anastasiia Ye, Jiayuan Bethune, Louis Katharopoulos, Angelos Grangier, David Computation and Language Machine Learning Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard training recipe is a two-stage paradigm: pretraining first on the full corpus of data followed by specialization on a subset of high quality, specialized data from the full corpus. In the multi-domain setting, this involves continued pretraining of multiple models on each specialized domain, referred to as split model training. We propose a method for pretraining multiple models independently over a general pretraining corpus, and determining the optimal compute allocation between pretraining and continued pretraining using scaling laws. Our approach accurately predicts the loss of a model of size N with D pretraining and D' specialization tokens, and extrapolates to larger model sizes and number of tokens. Applied to language model training, our approach improves performance consistently across common sense knowledge and reasoning benchmarks across different model sizes and compute budgets. |
| title | Optimal Splitting of Language Models from Mixtures to Specialized Domains |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2603.19149 |