Characterization of Large Language Model Development in the Datacenter
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917629813325824 |
|---|---|
| author | Hu, Qinghao Ye, Zhisheng Wang, Zerui Wang, Guoteng Zhang, Meng Chen, Qiaoling Sun, Peng Lin, Dahua Wang, Xiaolin Luo, Yingwei Wen, Yonggang Zhang, Tianwei |
| author_facet | Hu, Qinghao Ye, Zhisheng Wang, Zerui Wang, Guoteng Zhang, Meng Chen, Qiaoling Sun, Peng Lin, Dahua Wang, Xiaolin Luo, Yingwei Wen, Yonggang Zhang, Tianwei |
| contents | Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs, often riddled with numerous challenges such as frequent hardware failures, intricate parallelization strategies, and imbalanced resource utilization. In this paper, we present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme. Specifically, we investigate discrepancies between LLMs and prior task-specific Deep Learning (DL) workloads, explore resource utilization patterns, and identify the impact of various job failures. Our analysis summarizes hurdles we encountered and uncovers potential opportunities to optimize systems tailored for LLMs. Furthermore, we introduce our system efforts: (1) fault-tolerant pretraining, which enhances fault tolerance through LLM-involved failure diagnosis and automatic recovery. (2) decoupled scheduling for evaluation, which achieves timely performance feedback via trial decomposition and scheduling optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07648 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Characterization of Large Language Model Development in the Datacenter Hu, Qinghao Ye, Zhisheng Wang, Zerui Wang, Guoteng Zhang, Meng Chen, Qiaoling Sun, Peng Lin, Dahua Wang, Xiaolin Luo, Yingwei Wen, Yonggang Zhang, Tianwei Distributed, Parallel, and Cluster Computing Machine Learning Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs, often riddled with numerous challenges such as frequent hardware failures, intricate parallelization strategies, and imbalanced resource utilization. In this paper, we present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme. Specifically, we investigate discrepancies between LLMs and prior task-specific Deep Learning (DL) workloads, explore resource utilization patterns, and identify the impact of various job failures. Our analysis summarizes hurdles we encountered and uncovers potential opportunities to optimize systems tailored for LLMs. Furthermore, we introduce our system efforts: (1) fault-tolerant pretraining, which enhances fault tolerance through LLM-involved failure diagnosis and automatic recovery. (2) decoupled scheduling for evaluation, which achieves timely performance feedback via trial decomposition and scheduling optimization. |
| title | Characterization of Large Language Model Development in the Datacenter |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2403.07648 |