The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912276237254656 |
|---|---|
| author | Jin, Tian Humayun, Ahmed Imtiaz Evci, Utku Subramanian, Suvinay Yazdanbakhsh, Amir Alistarh, Dan Dziugaite, Gintare Karolina |
| author_facet | Jin, Tian Humayun, Ahmed Imtiaz Evci, Utku Subramanian, Suvinay Yazdanbakhsh, Amir Alistarh, Dan Dziugaite, Gintare Karolina |
| contents | Pruning eliminates unnecessary parameters in neural networks; it offers a promising solution to the growing computational demands of large language models (LLMs). While many focus on post-training pruning, sparse pre-training--which combines pruning and pre-training into a single phase--provides a simpler alternative. In this work, we present the first systematic exploration of optimal sparse pre-training configurations for LLMs through an examination of 80 unique pruning schedules across different sparsity levels and training durations. We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss. These findings provide valuable insights for efficient and effective sparse pre-training of LLMs. Furthermore, we propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training. Through empirical and theoretical validation, we demonstrate that this modified scaling law accurately models evaluation loss for both sparsely and densely pre-trained LLMs, unifying scaling laws across pre-training paradigms. Our findings indicate that while sparse pre-training achieves the same final model quality as dense pre-training for equivalent compute budgets, it provides substantial benefits through reduced model size, enabling significant potential computational savings during inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_12486 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws Jin, Tian Humayun, Ahmed Imtiaz Evci, Utku Subramanian, Suvinay Yazdanbakhsh, Amir Alistarh, Dan Dziugaite, Gintare Karolina Machine Learning Computation and Language Pruning eliminates unnecessary parameters in neural networks; it offers a promising solution to the growing computational demands of large language models (LLMs). While many focus on post-training pruning, sparse pre-training--which combines pruning and pre-training into a single phase--provides a simpler alternative. In this work, we present the first systematic exploration of optimal sparse pre-training configurations for LLMs through an examination of 80 unique pruning schedules across different sparsity levels and training durations. We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss. These findings provide valuable insights for efficient and effective sparse pre-training of LLMs. Furthermore, we propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training. Through empirical and theoretical validation, we demonstrate that this modified scaling law accurately models evaluation loss for both sparsely and densely pre-trained LLMs, unifying scaling laws across pre-training paradigms. Our findings indicate that while sparse pre-training achieves the same final model quality as dense pre-training for equivalent compute budgets, it provides substantial benefits through reduced model size, enabling significant potential computational savings during inference. |
| title | The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2501.12486 |