Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916318237687808 |
|---|---|
| author | Parmar, Jupinder Satheesh, Sanjev Patwary, Mostofa Shoeybi, Mohammad Catanzaro, Bryan |
| author_facet | Parmar, Jupinder Satheesh, Sanjev Patwary, Mostofa Shoeybi, Mohammad Catanzaro, Bryan |
| contents | As language models have scaled both their number of parameters and pretraining dataset sizes, the computational cost for pretraining has become intractable except for the most well-resourced teams. This increasing cost makes it ever more important to be able to reuse a model after it has completed pretraining; allowing for a model's abilities to further improve without needing to train from scratch. In this work, we detail a set of guidelines that cover how to design efficacious data distributions and learning rate schedules for continued pretraining of language models. When applying these findings within a continued pretraining run on top of a well-trained 15B parameter model, we show an improvement of 9\% in average model accuracy compared to the baseline of continued training on the pretraining set. The resulting recipe provides a practical starting point with which to begin developing language models through reuse rather than retraining. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07263 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models Parmar, Jupinder Satheesh, Sanjev Patwary, Mostofa Shoeybi, Mohammad Catanzaro, Bryan Computation and Language As language models have scaled both their number of parameters and pretraining dataset sizes, the computational cost for pretraining has become intractable except for the most well-resourced teams. This increasing cost makes it ever more important to be able to reuse a model after it has completed pretraining; allowing for a model's abilities to further improve without needing to train from scratch. In this work, we detail a set of guidelines that cover how to design efficacious data distributions and learning rate schedules for continued pretraining of language models. When applying these findings within a continued pretraining run on top of a well-trained 15B parameter model, we show an improvement of 9\% in average model accuracy compared to the baseline of continued training on the pretraining set. The resulting recipe provides a practical starting point with which to begin developing language models through reuse rather than retraining. |
| title | Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.07263 |