Overtrained Language Models Are Harder to Fine-Tune
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909555725697024 |
|---|---|
| author | Springer, Jacob Mitchell Goyal, Sachin Wen, Kaiyue Kumar, Tanishq Yue, Xiang Malladi, Sadhika Neubig, Graham Raghunathan, Aditi |
| author_facet | Springer, Jacob Mitchell Goyal, Sachin Wen, Kaiyue Kumar, Tanishq Yue, Xiang Malladi, Sadhika Neubig, Graham Raghunathan, Aditi |
| contents | Large language models are pre-trained on ever-growing token budgets under the assumption that better pre-training performance translates to improved downstream models. In this work, we challenge this assumption and show that extended pre-training can make models harder to fine-tune, leading to degraded final performance. We term this phenomenon catastrophic overtraining. For example, the instruction-tuned OLMo-1B model pre-trained on 3T tokens leads to over 2% worse performance on multiple standard LLM benchmarks than its 2.3T token counterpart. Through controlled experiments and theoretical analysis, we show that catastrophic overtraining arises from a systematic increase in the broad sensitivity of pre-trained parameters to modifications, including but not limited to fine-tuning. Our findings call for a critical reassessment of pre-training design that considers the downstream adaptability of the model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_19206 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Overtrained Language Models Are Harder to Fine-Tune Springer, Jacob Mitchell Goyal, Sachin Wen, Kaiyue Kumar, Tanishq Yue, Xiang Malladi, Sadhika Neubig, Graham Raghunathan, Aditi Computation and Language Artificial Intelligence Large language models are pre-trained on ever-growing token budgets under the assumption that better pre-training performance translates to improved downstream models. In this work, we challenge this assumption and show that extended pre-training can make models harder to fine-tune, leading to degraded final performance. We term this phenomenon catastrophic overtraining. For example, the instruction-tuned OLMo-1B model pre-trained on 3T tokens leads to over 2% worse performance on multiple standard LLM benchmarks than its 2.3T token counterpart. Through controlled experiments and theoretical analysis, we show that catastrophic overtraining arises from a systematic increase in the broad sensitivity of pre-trained parameters to modifications, including but not limited to fine-tuning. Our findings call for a critical reassessment of pre-training design that considers the downstream adaptability of the model. |
| title | Overtrained Language Models Are Harder to Fine-Tune |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.19206 |