Re-evaluating the Memory-balanced Pipeline Parallelism: BPipe
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909060997054464 |
|---|---|
| author | Huang, Mincong Wang, Chao Ma, Chi Zhang, Yineng Zhang, Peng Yu, Lei |
| author_facet | Huang, Mincong Wang, Chao Ma, Chi Zhang, Yineng Zhang, Peng Yu, Lei |
| contents | Pipeline parallelism is an essential technique in the training of large-scale Transformer models. However, it suffers from imbalanced memory consumption, leading to insufficient memory utilization. The BPipe technique was proposed to address this issue and has proven effective in the GPT-3 model. Nevertheless, our experiments have not yielded similar benefits for LLaMA training. Additionally, BPipe only yields negligible benefits for GPT-3 training when applying flash attention. We analyze the underlying causes of the divergent performance of BPipe on GPT-3 and LLaMA. Furthermore, we introduce a novel method to estimate the performance of BPipe. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02088 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Re-evaluating the Memory-balanced Pipeline Parallelism: BPipe Huang, Mincong Wang, Chao Ma, Chi Zhang, Yineng Zhang, Peng Yu, Lei Machine Learning Computation and Language Distributed, Parallel, and Cluster Computing Pipeline parallelism is an essential technique in the training of large-scale Transformer models. However, it suffers from imbalanced memory consumption, leading to insufficient memory utilization. The BPipe technique was proposed to address this issue and has proven effective in the GPT-3 model. Nevertheless, our experiments have not yielded similar benefits for LLaMA training. Additionally, BPipe only yields negligible benefits for GPT-3 training when applying flash attention. We analyze the underlying causes of the divergent performance of BPipe on GPT-3 and LLaMA. Furthermore, we introduce a novel method to estimate the performance of BPipe. |
| title | Re-evaluating the Memory-balanced Pipeline Parallelism: BPipe |
| topic | Machine Learning Computation and Language Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2401.02088 |