PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908368170385408 |
|---|---|
| author | van der Wal, Oskar Lesci, Pietro Muller-Eberstein, Max Saphra, Naomi Schoelkopf, Hailey Zuidema, Willem Biderman, Stella |
| author_facet | van der Wal, Oskar Lesci, Pietro Muller-Eberstein, Max Saphra, Naomi Schoelkopf, Hailey Zuidema, Willem Biderman, Stella |
| contents | The stability of language model pre-training and its effects on downstream performance are still understudied. Prior work shows that the training process can yield significantly different results in response to slight variations in initial conditions, e.g., the random seed. Crucially, the research community still lacks sufficient resources and tools to systematically investigate pre-training stability, particularly for decoder-only language models. We introduce the PolyPythias, a set of 45 new training runs for the Pythia model suite: 9 new seeds across 5 model sizes, from 14M to 410M parameters, resulting in about 7k new checkpoints that we release. Using these new 45 training runs, in addition to the 5 already available, we study the effects of different initial conditions determined by the seed -- i.e., parameters' initialisation and data order -- on (i) downstream performance, (ii) learned linguistic representations, and (iii) emergence of training phases. In addition to common scaling behaviours, our analyses generally reveal highly consistent training dynamics across both model sizes and initial conditions. Further, the new seeds for each model allow us to identify outlier training runs and delineate their characteristics. Our findings show the potential of using these methods to predict training stability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_09543 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs van der Wal, Oskar Lesci, Pietro Muller-Eberstein, Max Saphra, Naomi Schoelkopf, Hailey Zuidema, Willem Biderman, Stella Computation and Language Machine Learning The stability of language model pre-training and its effects on downstream performance are still understudied. Prior work shows that the training process can yield significantly different results in response to slight variations in initial conditions, e.g., the random seed. Crucially, the research community still lacks sufficient resources and tools to systematically investigate pre-training stability, particularly for decoder-only language models. We introduce the PolyPythias, a set of 45 new training runs for the Pythia model suite: 9 new seeds across 5 model sizes, from 14M to 410M parameters, resulting in about 7k new checkpoints that we release. Using these new 45 training runs, in addition to the 5 already available, we study the effects of different initial conditions determined by the seed -- i.e., parameters' initialisation and data order -- on (i) downstream performance, (ii) learned linguistic representations, and (iii) emergence of training phases. In addition to common scaling behaviours, our analyses generally reveal highly consistent training dynamics across both model sizes and initial conditions. Further, the new seeds for each model allow us to identify outlier training runs and delineate their characteristics. Our findings show the potential of using these methods to predict training stability. |
| title | PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.09543 |