Exploring Scaling Laws for Local SGD in Large Language Model Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912036589404160 |
|---|---|
| author | He, Qiaozhi Zhuang, Xiaomin Wu, Zhihua |
| author_facet | He, Qiaozhi Zhuang, Xiaomin Wu, Zhihua |
| contents | This paper investigates scaling laws for local SGD in LLM training, a distributed optimization algorithm that facilitates training on loosely connected devices. Through extensive experiments, we show that local SGD achieves competitive results compared to conventional methods, given equivalent model parameters, datasets, and computational resources. Furthermore, we explore the application of local SGD in various practical scenarios, including multi-cluster setups and edge computing environments. Our findings elucidate the necessary conditions for effective multi-cluster LLM training and examine the potential and limitations of leveraging edge computing resources in the LLM training process. This demonstrates its viability as an alternative to single large-cluster training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_13198 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring Scaling Laws for Local SGD in Large Language Model Training He, Qiaozhi Zhuang, Xiaomin Wu, Zhihua Computation and Language Machine Learning This paper investigates scaling laws for local SGD in LLM training, a distributed optimization algorithm that facilitates training on loosely connected devices. Through extensive experiments, we show that local SGD achieves competitive results compared to conventional methods, given equivalent model parameters, datasets, and computational resources. Furthermore, we explore the application of local SGD in various practical scenarios, including multi-cluster setups and edge computing environments. Our findings elucidate the necessary conditions for effective multi-cluster LLM training and examine the potential and limitations of leveraging edge computing resources in the LLM training process. This demonstrates its viability as an alternative to single large-cluster training. |
| title | Exploring Scaling Laws for Local SGD in Large Language Model Training |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2409.13198 |