Minder: Faulty Machine Detection for Large-scale Distributed Model Training
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912346777059328 |
|---|---|
| author | Deng, Yangtao Shi, Xiang Jiang, Zhuo Zhang, Xingjian Zhang, Lei Zhang, Zhang Li, Bo Song, Zuquan Zhu, Hang Liu, Gaohong Li, Fuliang Wang, Shuguang Lin, Haibin Ye, Jianxi Yu, Minlan |
| author_facet | Deng, Yangtao Shi, Xiang Jiang, Zhuo Zhang, Xingjian Zhang, Lei Zhang, Zhang Li, Bo Song, Zuquan Zhu, Hang Liu, Gaohong Li, Fuliang Wang, Shuguang Lin, Haibin Ye, Jianxi Yu, Minlan |
| contents | Large-scale distributed model training requires simultaneous training on up to thousands of machines. Faulty machine detection is critical when an unexpected fault occurs in a machine. From our experience, a training task can encounter two faults per day on average, possibly leading to a halt for hours. To address the drawbacks of the time-consuming and labor-intensive manual scrutiny, we propose Minder, an automatic faulty machine detector for distributed training tasks. The key idea of Minder is to automatically and efficiently detect faulty distinctive monitoring metric patterns, which could last for a period before the entire training task comes to a halt. Minder has been deployed in our production environment for over one year, monitoring daily distributed training tasks where each involves up to thousands of machines. In our real-world fault detection scenarios, Minder can accurately and efficiently react to faults within 3.6 seconds on average, with a precision of 0.904 and F1-score of 0.893. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_01791 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Minder: Faulty Machine Detection for Large-scale Distributed Model Training Deng, Yangtao Shi, Xiang Jiang, Zhuo Zhang, Xingjian Zhang, Lei Zhang, Zhang Li, Bo Song, Zuquan Zhu, Hang Liu, Gaohong Li, Fuliang Wang, Shuguang Lin, Haibin Ye, Jianxi Yu, Minlan Distributed, Parallel, and Cluster Computing Machine Learning Large-scale distributed model training requires simultaneous training on up to thousands of machines. Faulty machine detection is critical when an unexpected fault occurs in a machine. From our experience, a training task can encounter two faults per day on average, possibly leading to a halt for hours. To address the drawbacks of the time-consuming and labor-intensive manual scrutiny, we propose Minder, an automatic faulty machine detector for distributed training tasks. The key idea of Minder is to automatically and efficiently detect faulty distinctive monitoring metric patterns, which could last for a period before the entire training task comes to a halt. Minder has been deployed in our production environment for over one year, monitoring daily distributed training tasks where each involves up to thousands of machines. In our real-world fault detection scenarios, Minder can accurately and efficiently react to faults within 3.6 seconds on average, with a precision of 0.904 and F1-score of 0.893. |
| title | Minder: Faulty Machine Detection for Large-scale Distributed Model Training |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2411.01791 |