MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909220690984960 |
|---|---|
| author | Hsia, Samuel Golden, Alicia Acun, Bilge Ardalani, Newsha DeVito, Zachary Wei, Gu-Yeon Brooks, David Wu, Carole-Jean |
| author_facet | Hsia, Samuel Golden, Alicia Acun, Bilge Ardalani, Newsha DeVito, Zachary Wei, Gu-Yeon Brooks, David Wu, Carole-Jean |
| contents | Training and deploying large-scale machine learning models is time-consuming, requires significant distributed computing infrastructures, and incurs high operational costs. Our analysis, grounded in real-world large model training on datacenter-scale infrastructures, reveals that 14~32% of all GPU hours are spent on communication with no overlapping computation. To minimize this outstanding communication latency and other inherent at-scale inefficiencies, we introduce an agile performance modeling framework, MAD-Max. This framework is designed to optimize parallelization strategies and facilitate hardware-software co-design opportunities. Through the application of MAD-Max to a suite of real-world large-scale ML models on state-of-the-art GPU clusters, we showcase potential throughput enhancements of up to 2.24x for pre-training and up to 5.2x for inference scenarios, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_02784 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems Hsia, Samuel Golden, Alicia Acun, Bilge Ardalani, Newsha DeVito, Zachary Wei, Gu-Yeon Brooks, David Wu, Carole-Jean Distributed, Parallel, and Cluster Computing Hardware Architecture Machine Learning Training and deploying large-scale machine learning models is time-consuming, requires significant distributed computing infrastructures, and incurs high operational costs. Our analysis, grounded in real-world large model training on datacenter-scale infrastructures, reveals that 14~32% of all GPU hours are spent on communication with no overlapping computation. To minimize this outstanding communication latency and other inherent at-scale inefficiencies, we introduce an agile performance modeling framework, MAD-Max. This framework is designed to optimize parallelization strategies and facilitate hardware-software co-design opportunities. Through the application of MAD-Max to a suite of real-world large-scale ML models on state-of-the-art GPU clusters, we showcase potential throughput enhancements of up to 2.24x for pre-training and up to 5.2x for inference scenarios, respectively. |
| title | MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems |
| topic | Distributed, Parallel, and Cluster Computing Hardware Architecture Machine Learning |
| url | https://arxiv.org/abs/2310.02784 |