MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913243871576064 |
|---|---|
| author | Jiang, Ziheng Lin, Haibin Zhong, Yinmin Huang, Qi Chen, Yangrui Zhang, Zhi Peng, Yanghua Li, Xiang Xie, Cong Nong, Shibiao Jia, Yulu He, Sun Chen, Hongmin Bai, Zhihao Hou, Qi Yan, Shipeng Zhou, Ding Sheng, Yiyao Jiang, Zhuo Xu, Haohan Wei, Haoran Zhang, Zhang Nie, Pengfei Zou, Leqi Zhao, Sida Xiang, Liang Liu, Zherui Li, Zhe Jia, Xiaoying Ye, Jianxi Jin, Xin Liu, Xin |
| author_facet | Jiang, Ziheng Lin, Haibin Zhong, Yinmin Huang, Qi Chen, Yangrui Zhang, Zhi Peng, Yanghua Li, Xiang Xie, Cong Nong, Shibiao Jia, Yulu He, Sun Chen, Hongmin Bai, Zhihao Hou, Qi Yan, Shipeng Zhou, Ding Sheng, Yiyao Jiang, Zhuo Xu, Haohan Wei, Haoran Zhang, Zhang Nie, Pengfei Zou, Leqi Zhao, Sida Xiang, Liang Liu, Zherui Li, Zhe Jia, Xiaoying Ye, Jianxi Jin, Xin Liu, Xin |
| contents | We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high efficiency throughout the training process (i.e., stability) is an important consideration in production given the long extent of LLM training jobs. Many hard stability issues only emerge at large scale, and in-depth observability is the key to address them. We develop a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the MFU by 1.34x compared to Megatron-LM. We share our operational experience in identifying and fixing failures and stragglers. We hope by articulating the problems and sharing our experience from a systems perspective, this work can inspire future LLM systems research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_15627 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Jiang, Ziheng Lin, Haibin Zhong, Yinmin Huang, Qi Chen, Yangrui Zhang, Zhi Peng, Yanghua Li, Xiang Xie, Cong Nong, Shibiao Jia, Yulu He, Sun Chen, Hongmin Bai, Zhihao Hou, Qi Yan, Shipeng Zhou, Ding Sheng, Yiyao Jiang, Zhuo Xu, Haohan Wei, Haoran Zhang, Zhang Nie, Pengfei Zou, Leqi Zhao, Sida Xiang, Liang Liu, Zherui Li, Zhe Jia, Xiaoying Ye, Jianxi Jin, Xin Liu, Xin Machine Learning Distributed, Parallel, and Cluster Computing We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale brings unprecedented challenges to training efficiency and stability. We take a full-stack approach that co-designs the algorithmic and system components across model block and optimizer design, computation and communication overlapping, operator optimization, data pipeline, and network performance tuning. Maintaining high efficiency throughout the training process (i.e., stability) is an important consideration in production given the long extent of LLM training jobs. Many hard stability issues only emerge at large scale, and in-depth observability is the key to address them. We develop a set of diagnosis tools to monitor system components and events deep in the stack, identify root causes, and derive effective techniques to achieve fault tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the MFU by 1.34x compared to Megatron-LM. We share our operational experience in identifying and fixing failures and stragglers. We hope by articulating the problems and sharing our experience from a systems perspective, this work can inspire future LLM systems research. |
| title | MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2402.15627 |