Chameleon: Adaptive Fault Tolerance for Distributed Training via Real-time Policy Selection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918453775958016 |
|---|---|
| author | Zhou, Yuhang Wang, Zhibin Jiang, Peng Xia, Haoran Lu, Junhe Jiang, Qianyu Gu, Rong Xu, Hengxi Huang, Xinjing Fang, Guanghuan Hu, Zhiheng Zhang, Jingyi Cai, Yongjin He, Jian Tian, Chen |
| author_facet | Zhou, Yuhang Wang, Zhibin Jiang, Peng Xia, Haoran Lu, Junhe Jiang, Qianyu Gu, Rong Xu, Hengxi Huang, Xinjing Fang, Guanghuan Hu, Zhiheng Zhang, Jingyi Cai, Yongjin He, Jian Tian, Chen |
| contents | Training large language models faces frequent interruptions due to various faults, demanding robust fault-tolerance. Existing backup-free methods, such as redundant computation, dynamic parallelism, and data rerouting, each incur performance penalties, whether from ongoing overhead, lengthy reconfigurations, or post-recovery inefficiencies. We propose Chameleon, an adaptive fault-tolerant system that intelligently selects optimal recovery strategies when a failure occurs. Chameleon achieves this through a unified performance model, expedient execution plan search, accurate performance estimation, and efficient communication optimizations. Experiments on a 32-card cluster show that Chameleon maintains a performance gap of within 11.00% between post-recovery and failure-free training, while preserving model convergence and efficient memory usage. Compared to state-of-the-art methods, Chameleon achieves up to 1.229x and 1.355x higher average throughput than Oobleck and Recycle, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_21613 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Chameleon: Adaptive Fault Tolerance for Distributed Training via Real-time Policy Selection Zhou, Yuhang Wang, Zhibin Jiang, Peng Xia, Haoran Lu, Junhe Jiang, Qianyu Gu, Rong Xu, Hengxi Huang, Xinjing Fang, Guanghuan Hu, Zhiheng Zhang, Jingyi Cai, Yongjin He, Jian Tian, Chen Distributed, Parallel, and Cluster Computing Training large language models faces frequent interruptions due to various faults, demanding robust fault-tolerance. Existing backup-free methods, such as redundant computation, dynamic parallelism, and data rerouting, each incur performance penalties, whether from ongoing overhead, lengthy reconfigurations, or post-recovery inefficiencies. We propose Chameleon, an adaptive fault-tolerant system that intelligently selects optimal recovery strategies when a failure occurs. Chameleon achieves this through a unified performance model, expedient execution plan search, accurate performance estimation, and efficient communication optimizations. Experiments on a 32-card cluster show that Chameleon maintains a performance gap of within 11.00% between post-recovery and failure-free training, while preserving model convergence and efficient memory usage. Compared to state-of-the-art methods, Chameleon achieves up to 1.229x and 1.355x higher average throughput than Oobleck and Recycle, respectively. |
| title | Chameleon: Adaptive Fault Tolerance for Distributed Training via Real-time Policy Selection |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2508.21613 |