GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914992667754496 |
|---|---|
| author | Jeon, Byungsoo Wu, Mengdi Cao, Shiyi Kim, Sunghyun Park, Sunghyun Aggarwal, Neeraj Unger, Colin Arfeen, Daiyaan Liao, Peiyuan Miao, Xupeng Alizadeh, Mohammad Ganger, Gregory R. Chen, Tianqi Jia, Zhihao |
| author_facet | Jeon, Byungsoo Wu, Mengdi Cao, Shiyi Kim, Sunghyun Park, Sunghyun Aggarwal, Neeraj Unger, Colin Arfeen, Daiyaan Liao, Peiyuan Miao, Xupeng Alizadeh, Mohammad Ganger, Gregory R. Chen, Tianqi Jia, Zhihao |
| contents | Deep neural networks (DNNs) continue to grow rapidly in size, making them infeasible to train on a single device. Pipeline parallelism is commonly used in existing DNN systems to support large-scale DNN training by partitioning a DNN into multiple stages, which concurrently perform DNN training for different micro-batches in a pipeline fashion. However, existing pipeline-parallel approaches only consider sequential pipeline stages and thus ignore the topology of a DNN, resulting in missed model-parallel opportunities. This paper presents graph pipeline parallelism (GPP), a new pipeline-parallel scheme that partitions a DNN into pipeline stages whose dependencies are identified by a directed acyclic graph. GPP generalizes existing sequential pipeline parallelism and preserves the inherent topology of a DNN to enable concurrent execution of computationally-independent operators, resulting in reduced memory requirement and improved GPU performance. In addition, we develop GraphPipe, a distributed system that exploits GPP strategies to enable performant and scalable DNN training. GraphPipe partitions a DNN into a graph of stages, optimizes micro-batch schedules for these stages, and parallelizes DNN training using the discovered GPP strategies. Evaluation on a variety of DNNs shows that GraphPipe outperforms existing pipeline-parallel systems such as PipeDream and Piper by up to 1.6X. GraphPipe also reduces the search time by 9-21X compared to PipeDream and Piper. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_17145 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism Jeon, Byungsoo Wu, Mengdi Cao, Shiyi Kim, Sunghyun Park, Sunghyun Aggarwal, Neeraj Unger, Colin Arfeen, Daiyaan Liao, Peiyuan Miao, Xupeng Alizadeh, Mohammad Ganger, Gregory R. Chen, Tianqi Jia, Zhihao Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Deep neural networks (DNNs) continue to grow rapidly in size, making them infeasible to train on a single device. Pipeline parallelism is commonly used in existing DNN systems to support large-scale DNN training by partitioning a DNN into multiple stages, which concurrently perform DNN training for different micro-batches in a pipeline fashion. However, existing pipeline-parallel approaches only consider sequential pipeline stages and thus ignore the topology of a DNN, resulting in missed model-parallel opportunities. This paper presents graph pipeline parallelism (GPP), a new pipeline-parallel scheme that partitions a DNN into pipeline stages whose dependencies are identified by a directed acyclic graph. GPP generalizes existing sequential pipeline parallelism and preserves the inherent topology of a DNN to enable concurrent execution of computationally-independent operators, resulting in reduced memory requirement and improved GPU performance. In addition, we develop GraphPipe, a distributed system that exploits GPP strategies to enable performant and scalable DNN training. GraphPipe partitions a DNN into a graph of stages, optimizes micro-batch schedules for these stages, and parallelizes DNN training using the discovered GPP strategies. Evaluation on a variety of DNNs shows that GraphPipe outperforms existing pipeline-parallel systems such as PipeDream and Piper by up to 1.6X. GraphPipe also reduces the search time by 9-21X compared to PipeDream and Piper. |
| title | GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2406.17145 |