COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910366335762432 |
|---|---|
| author | Kadiyala, Divya Kiran Rashidi, Saeed Heo, Taekyung Bambhaniya, Abhimanyu Rajeshkumar Krishna, Tushar Daglis, Alexandros |
| author_facet | Kadiyala, Divya Kiran Rashidi, Saeed Heo, Taekyung Bambhaniya, Abhimanyu Rajeshkumar Krishna, Tushar Daglis, Alexandros |
| contents | Modern Deep Learning (DL) models have grown to sizes requiring massive clusters of specialized, high-end nodes to train. Designing such clusters to maximize both performance and utilization--to amortize their steep cost--is a challenging task requiring careful balance of compute, memory, and network resources. Moreover, a plethora of each model's tuning knobs drastically affect the performance, with optimal values often depending on the underlying cluster's characteristics, which necessitates a complex cluster-workload co-design process. To facilitate the design space exploration of such massive DL training clusters, we introduce COMET, a holistic cluster design methodology and workflow to jointly study the impact of parallelization strategies and key cluster resource provisioning on the performance of distributed DL training. We develop a step-by-step process to establish a reusable and flexible methodology, and demonstrate its application with case studies of training large models on cluster configurations of variable compute, memory, and network resources. Our case studies demonstrate COMET's utility in identifying promising architectural optimization directions and guiding system designers in configuring key model and cluster parameters. To illustrate, cluster configuration comparisons identify performance differences of up to 7.7x and highlight performance optimization opportunities of up to 1.4x when employing memory expansion as an optimization technique. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2211_16648 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Training Kadiyala, Divya Kiran Rashidi, Saeed Heo, Taekyung Bambhaniya, Abhimanyu Rajeshkumar Krishna, Tushar Daglis, Alexandros Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Modern Deep Learning (DL) models have grown to sizes requiring massive clusters of specialized, high-end nodes to train. Designing such clusters to maximize both performance and utilization--to amortize their steep cost--is a challenging task requiring careful balance of compute, memory, and network resources. Moreover, a plethora of each model's tuning knobs drastically affect the performance, with optimal values often depending on the underlying cluster's characteristics, which necessitates a complex cluster-workload co-design process. To facilitate the design space exploration of such massive DL training clusters, we introduce COMET, a holistic cluster design methodology and workflow to jointly study the impact of parallelization strategies and key cluster resource provisioning on the performance of distributed DL training. We develop a step-by-step process to establish a reusable and flexible methodology, and demonstrate its application with case studies of training large models on cluster configurations of variable compute, memory, and network resources. Our case studies demonstrate COMET's utility in identifying promising architectural optimization directions and guiding system designers in configuring key model and cluster parameters. To illustrate, cluster configuration comparisons identify performance differences of up to 7.7x and highlight performance optimization opportunities of up to 1.4x when employing memory expansion as an optimization technique. |
| title | COMET: A Comprehensive Cluster Design Methodology for Distributed Deep Learning Training |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2211.16648 |