FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Training
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917805020938240 |
|---|---|
| author | Wu, Tianyuan Wang, Wei Yu, Yinghao Yang, Siran Wu, Wenchao Duan, Qinkai Yang, Guodong Wang, Jiamang Qu, Lin Zhang, Liping |
| author_facet | Wu, Tianyuan Wang, Wei Yu, Yinghao Yang, Siran Wu, Wenchao Duan, Qinkai Yang, Guodong Wang, Jiamang Qu, Lin Zhang, Liping |
| contents | Fail-slows, or stragglers, are common but largely unheeded problems in large-scale hybrid-parallel training that spans thousands of GPU servers and runs for weeks to months. Yet, these problems are not well studied, nor can they be quickly detected and effectively mitigated. In this paper, we first present a characterization study on a shared production cluster with over 10,000 GPUs1. We find that fail-slows are caused by various CPU/GPU computation and cross-node networking issues, lasting from tens of seconds to nearly ten hours, and collectively delaying the average job completion time by 1.34%. The current practice is to manually detect these fail-slows and simply treat them as fail-stops using a checkpoint-and-restart failover approach, which are labor-intensive and time-consuming. In this paper, we propose FALCON, a framework that rapidly identifies fail-slowed GPUs and/or communication links, and effectively tackles them with a novel multi-level mitigation mechanism, all without human intervention. We have applied FALCON to detect human-labeled fail-slows in a production cluster with over 99% accuracy. Cluster deployment further demonstrates that FALCON effectively handles manually injected fail-slows, mitigating the training slowdown by 60.1%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12588 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Training Wu, Tianyuan Wang, Wei Yu, Yinghao Yang, Siran Wu, Wenchao Duan, Qinkai Yang, Guodong Wang, Jiamang Qu, Lin Zhang, Liping Distributed, Parallel, and Cluster Computing Operating Systems Fail-slows, or stragglers, are common but largely unheeded problems in large-scale hybrid-parallel training that spans thousands of GPU servers and runs for weeks to months. Yet, these problems are not well studied, nor can they be quickly detected and effectively mitigated. In this paper, we first present a characterization study on a shared production cluster with over 10,000 GPUs1. We find that fail-slows are caused by various CPU/GPU computation and cross-node networking issues, lasting from tens of seconds to nearly ten hours, and collectively delaying the average job completion time by 1.34%. The current practice is to manually detect these fail-slows and simply treat them as fail-stops using a checkpoint-and-restart failover approach, which are labor-intensive and time-consuming. In this paper, we propose FALCON, a framework that rapidly identifies fail-slowed GPUs and/or communication links, and effectively tackles them with a novel multi-level mitigation mechanism, all without human intervention. We have applied FALCON to detect human-labeled fail-slows in a production cluster with over 99% accuracy. Cluster deployment further demonstrates that FALCON effectively handles manually injected fail-slows, mitigating the training slowdown by 60.1%. |
| title | FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Training |
| topic | Distributed, Parallel, and Cluster Computing Operating Systems |
| url | https://arxiv.org/abs/2410.12588 |