Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2510.00606 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911196826828800 |
|---|---|
| author | Kang, Xueze Xiang, Guangyu Wang, Yuxin Zhang, Hao Fang, Yuchu Zhou, Yuhang Tang, Zhenheng Lv, Youhui Maman, Eliran Wasserman, Mark Zameret, Alon Bian, Zhipeng Chen, Shushu Yu, Zhiyou Wang, Jin Wu, Xiaoyu Zheng, Yang Tian, Chen Chu, Xiaowen |
| author_facet | Kang, Xueze Xiang, Guangyu Wang, Yuxin Zhang, Hao Fang, Yuchu Zhou, Yuhang Tang, Zhenheng Lv, Youhui Maman, Eliran Wasserman, Mark Zameret, Alon Bian, Zhipeng Chen, Shushu Yu, Zhiyou Wang, Jin Wu, Xiaoyu Zheng, Yang Tian, Chen Chu, Xiaowen |
| contents | Large-scale LLM pretraining now runs across $10^5$--$10^6$ accelerators, making failures routine and elasticity mandatory. We posit that an elastic-native training system must jointly deliver (i) parameter consistency, (ii) low mean time to recovery (MTTR), (iii) high post-change throughput, and (iv) computation consistency. No prior system achieves all four simultaneously. To achieve these goals, we present ElasWave, which delivers per-step fault tolerance via multi-dimensional scheduling across graph, dataflow, DVFS, and RNG. ElasWave reshapes and reshards micro-batches while preserving the global batch size and gradient scale. It performs online pipeline resharding with asynchronous parameter migration and interleaves ZeRO partitions, reducing parameter recovery processes to disjoint rank-to-rank transfers. It further leverages DVFS to absorb pipeline bubbles and reshards RNG to keep computation consistency. Together, a dynamic communicator enables in-place communication group edits, while per-step in-memory snapshots support online verification and redistribution. We evaluate ElasWave on 96 NPUs and benchmark it against state-of-the-art baselines: throughput improves by $1.35\times$ over ReCycle and $1.60\times$ over TorchFT; communicator recovery completes within one second (up to $82\times/3.6\times$ faster than full/partial rebuilds); migration MTTR drops by as much as $51\%$; and convergence deviation is reduced by approximately $78\%$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00606 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training Kang, Xueze Xiang, Guangyu Wang, Yuxin Zhang, Hao Fang, Yuchu Zhou, Yuhang Tang, Zhenheng Lv, Youhui Maman, Eliran Wasserman, Mark Zameret, Alon Bian, Zhipeng Chen, Shushu Yu, Zhiyou Wang, Jin Wu, Xiaoyu Zheng, Yang Tian, Chen Chu, Xiaowen Distributed, Parallel, and Cluster Computing Large-scale LLM pretraining now runs across $10^5$--$10^6$ accelerators, making failures routine and elasticity mandatory. We posit that an elastic-native training system must jointly deliver (i) parameter consistency, (ii) low mean time to recovery (MTTR), (iii) high post-change throughput, and (iv) computation consistency. No prior system achieves all four simultaneously. To achieve these goals, we present ElasWave, which delivers per-step fault tolerance via multi-dimensional scheduling across graph, dataflow, DVFS, and RNG. ElasWave reshapes and reshards micro-batches while preserving the global batch size and gradient scale. It performs online pipeline resharding with asynchronous parameter migration and interleaves ZeRO partitions, reducing parameter recovery processes to disjoint rank-to-rank transfers. It further leverages DVFS to absorb pipeline bubbles and reshards RNG to keep computation consistency. Together, a dynamic communicator enables in-place communication group edits, while per-step in-memory snapshots support online verification and redistribution. We evaluate ElasWave on 96 NPUs and benchmark it against state-of-the-art baselines: throughput improves by $1.35\times$ over ReCycle and $1.60\times$ over TorchFT; communicator recovery completes within one second (up to $82\times/3.6\times$ faster than full/partial rebuilds); migration MTTR drops by as much as $51\%$; and convergence deviation is reduced by approximately $78\%$. |
| title | ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2510.00606 |