Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912395767578624 |
|---|---|
| author | Bethune, Louis Grangier, David Busbridge, Dan Gualdoni, Eleonora Cuturi, Marco Ablin, Pierre |
| author_facet | Bethune, Louis Grangier, David Busbridge, Dan Gualdoni, Eleonora Cuturi, Marco Ablin, Pierre |
| contents | A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from that target domain. Finetuning presents two challenges: (i) if the amount of target data is limited, as in most practical applications, the model will quickly overfit, and (ii) the model will drift away from the original model, forgetting the pretraining data and the generic knowledge that comes with it. We aim to derive scaling laws that quantify these two phenomena for various target domains, amounts of available target data, and model scales. We measure the efficiency of injecting pretraining data into the finetuning data mixture to avoid forgetting and mitigate overfitting. A key practical takeaway from our study is that injecting as little as 1% of pretraining data in the finetuning data mixture prevents the model from forgetting the pretraining set. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06042 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection Bethune, Louis Grangier, David Busbridge, Dan Gualdoni, Eleonora Cuturi, Marco Ablin, Pierre Machine Learning Computation and Language A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from that target domain. Finetuning presents two challenges: (i) if the amount of target data is limited, as in most practical applications, the model will quickly overfit, and (ii) the model will drift away from the original model, forgetting the pretraining data and the generic knowledge that comes with it. We aim to derive scaling laws that quantify these two phenomena for various target domains, amounts of available target data, and model scales. We measure the efficiency of injecting pretraining data into the finetuning data mixture to avoid forgetting and mitigate overfitting. A key practical takeaway from our study is that injecting as little as 1% of pretraining data in the finetuning data mixture prevents the model from forgetting the pretraining set. |
| title | Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.06042 |