Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914955314331648 |
|---|---|
| author | Wang, Guanhua Zhang, Chengming Shen, Zheyu Li, Ang Ruwase, Olatunji |
| author_facet | Wang, Guanhua Zhang, Chengming Shen, Zheyu Li, Ang Ruwase, Olatunji |
| contents | Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs for parallelizing and accelerating the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces training and provides generic strategy of fine-grained communication and computation overlapping. Extensive results show that, comparing with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15241 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping Wang, Guanhua Zhang, Chengming Shen, Zheyu Li, Ang Ruwase, Olatunji Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs for parallelizing and accelerating the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces training and provides generic strategy of fine-grained communication and computation overlapping. Extensive results show that, comparing with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs. |
| title | Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2409.15241 |