Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.03644 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911300122050560 |
|---|---|
| author | Zhao, Bohan Wang, Yuanhong Liu, Chenglin Pan, Jiagi Yang, Guang Liu, Ruitao Zhang, Tingrui Luo, Kai Xu, Wei |
| author_facet | Zhao, Bohan Wang, Yuanhong Liu, Chenglin Pan, Jiagi Yang, Guang Liu, Ruitao Zhang, Tingrui Luo, Kai Xu, Wei |
| contents | Recent developments in large language models (LLMs) have introduced new requirements for efficient and robust training. As LLM clusters scale, node failures, lengthy recoveries, and bulky checkpoints erode efficiency. Infrequent asynchronous checkpoints trigger costly rollbacks, yet higher frequencies add prohibitive overhead. To address these challenges, we propose FFTrainer, a system designed for robust LLM training. FFTrainer leverages surplus network capacity to quickly save and load states, thereby preventing rollbacks and accelerating recovery. Compared with prior checkpointing approaches, FFTrainer reduces recovery time by up to 98% and mitigates GPU utilization loss by up to 68% without hindering normal training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_03644 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FFTrainer: Fast Failover in Large-Language Model Training with Almost-Free State Management Zhao, Bohan Wang, Yuanhong Liu, Chenglin Pan, Jiagi Yang, Guang Liu, Ruitao Zhang, Tingrui Luo, Kai Xu, Wei Distributed, Parallel, and Cluster Computing Recent developments in large language models (LLMs) have introduced new requirements for efficient and robust training. As LLM clusters scale, node failures, lengthy recoveries, and bulky checkpoints erode efficiency. Infrequent asynchronous checkpoints trigger costly rollbacks, yet higher frequencies add prohibitive overhead. To address these challenges, we propose FFTrainer, a system designed for robust LLM training. FFTrainer leverages surplus network capacity to quickly save and load states, thereby preventing rollbacks and accelerating recovery. Compared with prior checkpointing approaches, FFTrainer reduces recovery time by up to 98% and mitigates GPU utilization loss by up to 68% without hindering normal training. |
| title | FFTrainer: Fast Failover in Large-Language Model Training with Almost-Free State Management |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2512.03644 |