Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915555250798592 |
|---|---|
| author | Li, Haoyang Fu, Fangcheng Lin, Sheng Ge, Hao Wang, Xuanyu Niu, Jiawen Xue, Jinbao Tao, Yangyu Wang, Di Jiang, Jie Cui, Bin |
| author_facet | Li, Haoyang Fu, Fangcheng Lin, Sheng Ge, Hao Wang, Xuanyu Niu, Jiawen Xue, Jinbao Tao, Yangyu Wang, Di Jiang, Jie Cui, Bin |
| contents | To optimize large Transformer model training, both efficient parallel computing and advanced data management are indispensable. However, current methods often assume a stable and uniform training workload, neglecting data-induced imbalances-arising from both sampling and packing processes-which can impede training performance. Specifically, data sampling imbalance arises from uneven sequence length distribution of the training data, while data packing imbalance stems from the discrepancy between the linear memory complexity and quadratic time complexity of the attention mechanism. To address these imbalance issues, we develop Hydraulis, which jointly optimizes the parallel strategies and data assignment. For one thing, we introduce large model training with dynamic heterogeneous parallel strategies in response to the sequence length variations within and across training iterations. For another, we devise a two-stage data assignment approach, which strikes a good balance in terms of the training workloads both within and across model replicas. Empirical results demonstrate that Hydraulis outperforms existing systems by 1.32-2.66 times. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07894 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment Li, Haoyang Fu, Fangcheng Lin, Sheng Ge, Hao Wang, Xuanyu Niu, Jiawen Xue, Jinbao Tao, Yangyu Wang, Di Jiang, Jie Cui, Bin Distributed, Parallel, and Cluster Computing Machine Learning To optimize large Transformer model training, both efficient parallel computing and advanced data management are indispensable. However, current methods often assume a stable and uniform training workload, neglecting data-induced imbalances-arising from both sampling and packing processes-which can impede training performance. Specifically, data sampling imbalance arises from uneven sequence length distribution of the training data, while data packing imbalance stems from the discrepancy between the linear memory complexity and quadratic time complexity of the attention mechanism. To address these imbalance issues, we develop Hydraulis, which jointly optimizes the parallel strategies and data assignment. For one thing, we introduce large model training with dynamic heterogeneous parallel strategies in response to the sequence length variations within and across training iterations. For another, we devise a two-stage data assignment approach, which strikes a good balance in terms of the training workloads both within and across model replicas. Empirical results demonstrate that Hydraulis outperforms existing systems by 1.32-2.66 times. |
| title | Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2412.07894 |