When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913177051070464 |
|---|---|
| author | Wu, Boqian Xiao, Qiao Okanovic, Patrik Sternal, Tomasz van Keulen, Maurice Pechenizkiy, Mykola Mocanu, Elena Hoefler, Torsten Mocanu, Decebal Constantin |
| author_facet | Wu, Boqian Xiao, Qiao Okanovic, Patrik Sternal, Tomasz van Keulen, Maurice Pechenizkiy, Mykola Mocanu, Elena Hoefler, Torsten Mocanu, Decebal Constantin |
| contents | Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs; we further validate extrapolation on held-out dense-equivalent models up to 7.68B parameters. We find that: 1. Sparse scaling in data-limited settings: We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. 2. Delayed data saturation: sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. 3. Resource trade-offs: With fixed data, loss-optimal sparsity is moderate ~ 50%, while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity. Our code is available at: https://github.com/boqian333/sparse-dc-scaling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01155 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When Data Is Scarce: Scaling Sparse Language Models with Repeated Training Wu, Boqian Xiao, Qiao Okanovic, Patrik Sternal, Tomasz van Keulen, Maurice Pechenizkiy, Mykola Mocanu, Elena Hoefler, Torsten Mocanu, Decebal Constantin Machine Learning Artificial Intelligence Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs; we further validate extrapolation on held-out dense-equivalent models up to 7.68B parameters. We find that: 1. Sparse scaling in data-limited settings: We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. 2. Delayed data saturation: sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. 3. Resource trade-offs: With fixed data, loss-optimal sparsity is moderate ~ 50%, while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity. Our code is available at: https://github.com/boqian333/sparse-dc-scaling. |
| title | When Data Is Scarce: Scaling Sparse Language Models with Repeated Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2606.01155 |