Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909011620659200 |
|---|---|
| author | Watts, Ishaan Li, Catherine Goyal, Sachin Springer, Jacob Mitchell Raghunathan, Aditi |
| author_facet | Watts, Ishaan Li, Catherine Goyal, Sachin Springer, Jacob Mitchell Raghunathan, Aditi |
| contents | Pretraining optimizers are tuned to produce the strongest possible base model, on the assumption that a stronger starting point yields a stronger model after subsequent changes like post-training and quantization. This overlooks the geometry of the base model which controls how much of the base model's capabilities survive subsequent parameter updates. We study three pretraining optimization approaches that bias optimization toward flatter minima: Sharpness-Aware Minimization (SAM), large learning rates, and shortened learning rate annealing periods. Across model sizes ranging from 20M to 150M parameters, we find that these interventions consistently improve downstream performance after post-training on five common datasets with up to 80% less forgetting. These principles hold at scale: a short SAM mid-training phase applied to an existing OLMo-2-1B checkpoint reduces forgetting by 31% after MetaMath post-training and by 40% after 4-bit quantization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_02105 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting Watts, Ishaan Li, Catherine Goyal, Sachin Springer, Jacob Mitchell Raghunathan, Aditi Machine Learning Computation and Language Pretraining optimizers are tuned to produce the strongest possible base model, on the assumption that a stronger starting point yields a stronger model after subsequent changes like post-training and quantization. This overlooks the geometry of the base model which controls how much of the base model's capabilities survive subsequent parameter updates. We study three pretraining optimization approaches that bias optimization toward flatter minima: Sharpness-Aware Minimization (SAM), large learning rates, and shortened learning rate annealing periods. Across model sizes ranging from 20M to 150M parameters, we find that these interventions consistently improve downstream performance after post-training on five common datasets with up to 80% less forgetting. These principles hold at scale: a short SAM mid-training phase applied to an existing OLMo-2-1B checkpoint reduces forgetting by 31% after MetaMath post-training and by 40% after 4-bit quantization. |
| title | Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2605.02105 |