Training DNN Models over Heterogeneous Clusters with Optimal Performance
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909098842259456 |
|---|---|
| author | Nie, Chengyi Maghakian, Jessica Liu, Zhenhua |
| author_facet | Nie, Chengyi Maghakian, Jessica Liu, Zhenhua |
| contents | Adjusting batch sizes and adaptively tuning other hyperparameters can significantly speed up deep neural network (DNN) training. Despite the ubiquity of heterogeneous clusters, existing adaptive DNN training techniques solely consider homogeneous environments. Optimizing distributed DNN training over heterogeneous clusters is technically challenging, and directly adapting existing techniques results in low utilization and poor performance. To solve this problem, we introduce Cannikin -- a novel data-parallel distributed training system. Cannikin achieves efficient and near-optimal performance by accurately modeling the optimal system performance and predicting adaptive batch size training metrics for DNNs in heterogeneous clusters. We implemented Cannikin in PyTorch and conducted experiments over 16 GPUs in Chameleon. Empirical results show that Cannikin reduces DNN training in heterogeneous clusters by up to $52\%$ compared to the state-of-the-art adaptive training system and up to $85\%$ compared to native PyTorch DistributedDataParallel. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05302 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Training DNN Models over Heterogeneous Clusters with Optimal Performance Nie, Chengyi Maghakian, Jessica Liu, Zhenhua Distributed, Parallel, and Cluster Computing Adjusting batch sizes and adaptively tuning other hyperparameters can significantly speed up deep neural network (DNN) training. Despite the ubiquity of heterogeneous clusters, existing adaptive DNN training techniques solely consider homogeneous environments. Optimizing distributed DNN training over heterogeneous clusters is technically challenging, and directly adapting existing techniques results in low utilization and poor performance. To solve this problem, we introduce Cannikin -- a novel data-parallel distributed training system. Cannikin achieves efficient and near-optimal performance by accurately modeling the optimal system performance and predicting adaptive batch size training metrics for DNNs in heterogeneous clusters. We implemented Cannikin in PyTorch and conducted experiments over 16 GPUs in Chameleon. Empirical results show that Cannikin reduces DNN training in heterogeneous clusters by up to $52\%$ compared to the state-of-the-art adaptive training system and up to $85\%$ compared to native PyTorch DistributedDataParallel. |
| title | Training DNN Models over Heterogeneous Clusters with Optimal Performance |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2402.05302 |