Demystifying the Communication Characteristics for Distributed Transformer Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914916744560640 |
|---|---|
| author | Anthony, Quentin Michalowicz, Benjamin Hatef, Jacob Xu, Lang Abduljabbar, Mustafa Shafi, Aamir Subramoni, Hari Panda, Dhabaleswar |
| author_facet | Anthony, Quentin Michalowicz, Benjamin Hatef, Jacob Xu, Lang Abduljabbar, Mustafa Shafi, Aamir Subramoni, Hari Panda, Dhabaleswar |
| contents | Deep learning (DL) models based on the transformer architecture have revolutionized many DL applications such as large language models (LLMs), vision transformers, audio generation, and time series prediction. Much of this progress has been fueled by distributed training, yet distributed communication remains a substantial bottleneck to training progress. This paper examines the communication behavior of transformer models - that is, how different parallelism schemes used in multi-node/multi-GPU DL Training communicate data in the context of transformers. We use GPT-based language models as a case study of the transformer architecture due to their ubiquity. We validate the empirical results obtained from our communication logs using analytical models. At a high level, our analysis reveals a need to optimize small message point-to-point communication further, correlations between sequence length, per-GPU throughput, model size, and optimizations used, and where to potentially guide further optimizations in framework and HPC middleware design and optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10197 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Demystifying the Communication Characteristics for Distributed Transformer Models Anthony, Quentin Michalowicz, Benjamin Hatef, Jacob Xu, Lang Abduljabbar, Mustafa Shafi, Aamir Subramoni, Hari Panda, Dhabaleswar Distributed, Parallel, and Cluster Computing Artificial Intelligence Deep learning (DL) models based on the transformer architecture have revolutionized many DL applications such as large language models (LLMs), vision transformers, audio generation, and time series prediction. Much of this progress has been fueled by distributed training, yet distributed communication remains a substantial bottleneck to training progress. This paper examines the communication behavior of transformer models - that is, how different parallelism schemes used in multi-node/multi-GPU DL Training communicate data in the context of transformers. We use GPT-based language models as a case study of the transformer architecture due to their ubiquity. We validate the empirical results obtained from our communication logs using analytical models. At a high level, our analysis reveals a need to optimize small message point-to-point communication further, correlations between sequence length, per-GPU throughput, model size, and optimizations used, and where to potentially guide further optimizations in framework and HPC middleware design and optimization. |
| title | Demystifying the Communication Characteristics for Distributed Transformer Models |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2408.10197 |