Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.00254 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912174607171584 |
|---|---|
| author | Li, Zongbiao Li, Xiezhao Cui, Yinghao Chen, Yijun Gu, Zhixuan Liu, Yuxuan Zhu, Wenbo Jia, Fei Liu, Ke Li, Qifeng Zhan, Junyao Zhou, Jiangtao Zhang, Chenxi Liu, Qike |
| author_facet | Li, Zongbiao Li, Xiezhao Cui, Yinghao Chen, Yijun Gu, Zhixuan Liu, Yuxuan Zhu, Wenbo Jia, Fei Liu, Ke Li, Qifeng Zhan, Junyao Zhou, Jiangtao Zhang, Chenxi Liu, Qike |
| contents | The number of parameters in large-scale language models based on transformers is gradually increasing, and the scale of computing clusters is also growing. The technology of quickly mobilizing large amounts of computing resources for parallel computing is becoming increasingly important. In this paper, we propose an automatic parallel algorithm that automatically plans the parallel strategy with maximum throughput based on model and hardware information. By decoupling the training time into computation, communication, and overlap, we established a training duration simulation model. Based on this simulation model, we prune the parallel solution space to shorten the search time required. The multi-node experiment results show that the algorithm can estimate the parallel training duration in real time with an average accuracy of 96%. In our test, the recommendation strategy provided by the algorithm is always globally optimal. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00254 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Automatically Planning Optimal Parallel Strategy for Large Language Models Li, Zongbiao Li, Xiezhao Cui, Yinghao Chen, Yijun Gu, Zhixuan Liu, Yuxuan Zhu, Wenbo Jia, Fei Liu, Ke Li, Qifeng Zhan, Junyao Zhou, Jiangtao Zhang, Chenxi Liu, Qike Artificial Intelligence Computation and Language The number of parameters in large-scale language models based on transformers is gradually increasing, and the scale of computing clusters is also growing. The technology of quickly mobilizing large amounts of computing resources for parallel computing is becoming increasingly important. In this paper, we propose an automatic parallel algorithm that automatically plans the parallel strategy with maximum throughput based on model and hardware information. By decoupling the training time into computation, communication, and overlap, we established a training duration simulation model. Based on this simulation model, we prune the parallel solution space to shorten the search time required. The multi-node experiment results show that the algorithm can estimate the parallel training duration in real time with an average accuracy of 96%. In our test, the recommendation strategy provided by the algorithm is always globally optimal. |
| title | Automatically Planning Optimal Parallel Strategy for Large Language Models |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2501.00254 |