Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.17821 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910229846818816 |
|---|---|
| author | Han, Shujie Jiang, Feng Lee, Patrick P. C. Zhang, Xiao Huang, Zhijie Zhao, Nannan Zhao, Xiaonan Pan, Lichen |
| author_facet | Han, Shujie Jiang, Feng Lee, Patrick P. C. Zhang, Xiao Huang, Zhijie Zhao, Nannan Zhao, Xiaonan Pan, Lichen |
| contents | Large Language Model (LLM) training is frequently interrupted by a heterogeneous spectrum of failures, from common GPU crashes to catastrophic cluster-wide outages. Existing checkpointing systems rely on monolithic, single-tier storage backend, forcing a trade-off between state-saving overhead and recovery speed. We propose TierCheck, a cluster-aware tiered checkpointing system that aligns storage placement with failure heterogeneity. TierCheck adopts a three-tier design that maintains lightweight differential checkpoints in local and peer memory for fast localized recovery, while asynchronously migrating heavyweight base checkpoints to remote persistent storage. It also ensures strict global consistency across tiers without stalling training, and achieves fast cluster-aware checkpoint restoration during recovery. Evaluations on models up to 40 billion parameters show that TierCheck achieves low training overhead, reduces end-to-end checkpointing time to under 10s, and supports high-frequency checkpointing, ultimately striking an optimal balance between low-overhead persistence and fast recovery. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_17821 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training Han, Shujie Jiang, Feng Lee, Patrick P. C. Zhang, Xiao Huang, Zhijie Zhao, Nannan Zhao, Xiaonan Pan, Lichen Distributed, Parallel, and Cluster Computing Artificial Intelligence Large Language Model (LLM) training is frequently interrupted by a heterogeneous spectrum of failures, from common GPU crashes to catastrophic cluster-wide outages. Existing checkpointing systems rely on monolithic, single-tier storage backend, forcing a trade-off between state-saving overhead and recovery speed. We propose TierCheck, a cluster-aware tiered checkpointing system that aligns storage placement with failure heterogeneity. TierCheck adopts a three-tier design that maintains lightweight differential checkpoints in local and peer memory for fast localized recovery, while asynchronously migrating heavyweight base checkpoints to remote persistent storage. It also ensures strict global consistency across tiers without stalling training, and achieves fast cluster-aware checkpoint restoration during recovery. Evaluations on models up to 40 billion parameters show that TierCheck achieves low training overhead, reduces end-to-end checkpointing time to under 10s, and supports high-frequency checkpointing, ultimately striking an optimal balance between low-overhead persistence and fast recovery. |
| title | TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2605.17821 |