Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913733573345280 |
|---|---|
| author | Charles, Zachary Teston, Gabriel Dery, Lucio Rush, Keith Fallen, Nova Garrett, Zachary Szlam, Arthur Douillard, Arthur |
| author_facet | Charles, Zachary Teston, Gabriel Dery, Lucio Rush, Keith Fallen, Nova Garrett, Zachary Szlam, Arthur Douillard, Arthur |
| contents | As we scale to more massive machine learning models, the frequent synchronization demands inherent in data-parallel approaches create significant slowdowns, posing a critical challenge to further scaling. Recent work develops an approach (DiLoCo) that relaxes synchronization demands without compromising model quality. However, these works do not carefully analyze how DiLoCo's behavior changes with model size. In this work, we study the scaling law behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on how algorithmic factors, including number of model replicas, hyperparameters, and token budget affect training in ways that can be accurately predicted via scaling laws. We find that DiLoCo scales both predictably and robustly with model size. When well-tuned, DiLoCo scales better than data-parallel training with model size, and can outperform data-parallel training even at small model sizes. Our results showcase a more general set of benefits of DiLoCo than previously documented, including increased optimal batch sizes, improved downstream generalization with scale, and improved evaluation loss for a fixed token budget. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_09799 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo Charles, Zachary Teston, Gabriel Dery, Lucio Rush, Keith Fallen, Nova Garrett, Zachary Szlam, Arthur Douillard, Arthur Machine Learning Computation and Language Distributed, Parallel, and Cluster Computing As we scale to more massive machine learning models, the frequent synchronization demands inherent in data-parallel approaches create significant slowdowns, posing a critical challenge to further scaling. Recent work develops an approach (DiLoCo) that relaxes synchronization demands without compromising model quality. However, these works do not carefully analyze how DiLoCo's behavior changes with model size. In this work, we study the scaling law behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on how algorithmic factors, including number of model replicas, hyperparameters, and token budget affect training in ways that can be accurately predicted via scaling laws. We find that DiLoCo scales both predictably and robustly with model size. When well-tuned, DiLoCo scales better than data-parallel training with model size, and can outperform data-parallel training even at small model sizes. Our results showcase a more general set of benefits of DiLoCo than previously documented, including increased optimal batch sizes, improved downstream generalization with scale, and improved evaluation loss for a fixed token budget. |
| title | Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo |
| topic | Machine Learning Computation and Language Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2503.09799 |