FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916930874507264 |
|---|---|
| author | Zhang, Haijun Wang, Jinxiang Yu, Zhenhua Zhang, Yanyong Ji, Xuejie Mao, Kaining Zhang, Jun Zhang, Yaqing Wu, Ting Jie, Fei Huang, Xiemin Cai, Zhifang Cheng, Junhua Wang, Shuwei Li, Wei Bao, Xiaoming Xu, Hua Zhao, Shixiong Li, Jun Sun, Hongwei Zhang, Ziyang Xiong, Yi Li, Chunsheng |
| author_facet | Zhang, Haijun Wang, Jinxiang Yu, Zhenhua Zhang, Yanyong Ji, Xuejie Mao, Kaining Zhang, Jun Zhang, Yaqing Wu, Ting Jie, Fei Huang, Xiemin Cai, Zhifang Cheng, Junhua Wang, Shuwei Li, Wei Bao, Xiaoming Xu, Hua Zhao, Shixiong Li, Jun Sun, Hongwei Zhang, Ziyang Xiong, Yi Li, Chunsheng |
| contents | Large language models (LLMs) have made a profound impact across various fields due to their advanced capabilities. However, training these models at unprecedented scales requires extensive AI accelerator clusters and sophisticated parallelism strategies, which pose significant challenges in maintaining system reliability over prolonged training periods. A major concern is the substantial loss of training time caused by inevitable hardware and software failures. To address these challenges, we present FlashRecovery, a fast and low-cost failure recovery system comprising three core modules: (1) Active and real-time failure detection. This module performs continuous training state monitoring, enabling immediate identification of hardware and software failures within seconds, thus ensuring rapid incident response; (2) Scale-independent task restart. By employing different recovery strategies for normal and faulty nodes, combined with an optimized communication group reconstruction protocol, our approach ensures that the recovery time remains nearly constant, regardless of cluster scale; (3) Checkpoint-free recovery within one step. Our novel recovery mechanism enables single-step restoration, completely eliminating dependence on traditional checkpointing methods and their associated overhead. Collectively, these innovations enable FlashRecovery to achieve optimal Recovery Time Objective (RTO) and Recovery Point Objective (RPO), substantially improving the reliability and efficiency of long-duration LLM training. Experimental results demonstrate that FlashRecovery system can achieve training restoration on training cluster with 4, 800 devices in 150 seconds. We also verify that the time required for failure recovery is nearly consistent for different scales of training tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_03047 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs Zhang, Haijun Wang, Jinxiang Yu, Zhenhua Zhang, Yanyong Ji, Xuejie Mao, Kaining Zhang, Jun Zhang, Yaqing Wu, Ting Jie, Fei Huang, Xiemin Cai, Zhifang Cheng, Junhua Wang, Shuwei Li, Wei Bao, Xiaoming Xu, Hua Zhao, Shixiong Li, Jun Sun, Hongwei Zhang, Ziyang Xiong, Yi Li, Chunsheng Distributed, Parallel, and Cluster Computing Artificial Intelligence Large language models (LLMs) have made a profound impact across various fields due to their advanced capabilities. However, training these models at unprecedented scales requires extensive AI accelerator clusters and sophisticated parallelism strategies, which pose significant challenges in maintaining system reliability over prolonged training periods. A major concern is the substantial loss of training time caused by inevitable hardware and software failures. To address these challenges, we present FlashRecovery, a fast and low-cost failure recovery system comprising three core modules: (1) Active and real-time failure detection. This module performs continuous training state monitoring, enabling immediate identification of hardware and software failures within seconds, thus ensuring rapid incident response; (2) Scale-independent task restart. By employing different recovery strategies for normal and faulty nodes, combined with an optimized communication group reconstruction protocol, our approach ensures that the recovery time remains nearly constant, regardless of cluster scale; (3) Checkpoint-free recovery within one step. Our novel recovery mechanism enables single-step restoration, completely eliminating dependence on traditional checkpointing methods and their associated overhead. Collectively, these innovations enable FlashRecovery to achieve optimal Recovery Time Objective (RTO) and Recovery Point Objective (RPO), substantially improving the reliability and efficiency of long-duration LLM training. Experimental results demonstrate that FlashRecovery system can achieve training restoration on training cluster with 4, 800 devices in 150 seconds. We also verify that the time required for failure recovery is nearly consistent for different scales of training tasks. |
| title | FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2509.03047 |