Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912106637426688 |
|---|---|
| author | Lau, Tim Tsz-Kit Li, Weijian Xu, Chenwei Liu, Han Kolar, Mladen |
| author_facet | Lau, Tim Tsz-Kit Li, Weijian Xu, Chenwei Liu, Han Kolar, Mladen |
| contents | Modern deep neural networks often require distributed training with many workers due to their large size. As the number of workers increases, communication overheads become the main bottleneck in data-parallel minibatch stochastic gradient methods with per-iteration gradient synchronization. Local gradient methods like Local SGD reduce communication by only synchronizing model parameters and/or gradients after several local steps. Despite an understanding of their convergence and the importance of batch sizes for training efficiency and generalization, optimal batch sizes for local gradient methods are difficult to determine. We introduce adaptive batch size strategies for local gradient methods that increase batch sizes adaptively to reduce minibatch gradient variance. We provide convergence guarantees under homogeneous data conditions and support our claims with image classification and language modeling experiments, demonstrating the effectiveness of our strategies for both training efficiency and generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13936 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods Lau, Tim Tsz-Kit Li, Weijian Xu, Chenwei Liu, Han Kolar, Mladen Machine Learning Optimization and Control Modern deep neural networks often require distributed training with many workers due to their large size. As the number of workers increases, communication overheads become the main bottleneck in data-parallel minibatch stochastic gradient methods with per-iteration gradient synchronization. Local gradient methods like Local SGD reduce communication by only synchronizing model parameters and/or gradients after several local steps. Despite an understanding of their convergence and the importance of batch sizes for training efficiency and generalization, optimal batch sizes for local gradient methods are difficult to determine. We introduce adaptive batch size strategies for local gradient methods that increase batch sizes adaptively to reduce minibatch gradient variance. We provide convergence guarantees under homogeneous data conditions and support our claims with image classification and language modeling experiments, demonstrating the effectiveness of our strategies for both training efficiency and generalization. |
| title | Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2406.13936 |