ZeroPP: Unleashing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913360522510336 |
|---|---|
| author | Tang, Ding Jiang, Lijuan Zhou, Jiecheng Jin, Minxi Li, Hengjie Zhang, Xingcheng Pei, Zhilin Zhai, Jidong |
| author_facet | Tang, Ding Jiang, Lijuan Zhou, Jiecheng Jin, Minxi Li, Hengjie Zhang, Xingcheng Pei, Zhilin Zhai, Jidong |
| contents | Large-scale models rely heavily on 3D parallelism for distributed training, which utilizes tensor parallelism (TP) as the intra-operator parallelism to partition model states across GPUs. However, TP introduces significant communication overheads and complexity in modifying single-GPU code. In this paper, we propose a TP-free distributed framework ZeroPP, which leverages the hybrid of scalable inter-operator pipeline parallelism and intra-operator fully sharded data parallelism to train models at scale, reducing memory consumption and enabling high training efficiency. Through extensive experimentation, we demonstrate that ZeroPP achieves significant performance gains of up to 33% compared to conventional 3D parallelism while maintaining comparable GPU memory consumption. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_03791 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ZeroPP: Unleashing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology Tang, Ding Jiang, Lijuan Zhou, Jiecheng Jin, Minxi Li, Hengjie Zhang, Xingcheng Pei, Zhilin Zhai, Jidong Distributed, Parallel, and Cluster Computing Large-scale models rely heavily on 3D parallelism for distributed training, which utilizes tensor parallelism (TP) as the intra-operator parallelism to partition model states across GPUs. However, TP introduces significant communication overheads and complexity in modifying single-GPU code. In this paper, we propose a TP-free distributed framework ZeroPP, which leverages the hybrid of scalable inter-operator pipeline parallelism and intra-operator fully sharded data parallelism to train models at scale, reducing memory consumption and enabling high training efficiency. Through extensive experimentation, we demonstrate that ZeroPP achieves significant performance gains of up to 33% compared to conventional 3D parallelism while maintaining comparable GPU memory consumption. |
| title | ZeroPP: Unleashing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2402.03791 |