DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910234375618560 |
|---|---|
| author | Wang, Yuanqing Zhang, Yuchen Lin, Hao Hu, Junhao Zhu, Chunyang Zhang, Quanlu Li, Boxun Dai, Guohao Yang, Zhi Cheng, Daning Zhang, Yunquan Wang, Yu |
| author_facet | Wang, Yuanqing Zhang, Yuchen Lin, Hao Hu, Junhao Zhu, Chunyang Zhang, Quanlu Li, Boxun Dai, Guohao Yang, Zhi Cheng, Daning Zhang, Yunquan Wang, Yu |
| contents | Modern large language model (LLM) training is inherently dynamic: resource fluctuations, RLHF phase shifts, and cluster elasticity continually reshape the optimal parallelism layout, posing a significant challenge to existing training frameworks built around a static execution model. We present DynaTrain, a distributed training system for sub-second, online reconfiguration across arbitrary multi-dimensional parallelism. At its core, we propose a Virtual Parameter Space (VPS) abstraction that unifies all distributed training states under one logical coordinate space, turning any parallelism configuration into a deterministic mapping and collapsing complex transition into manageable geometric intersections. On top of VPS, a state routing-and-transition layer executes rank-local transfers under a memory-aware, deadlock-free schedule, and an Elastic Device Manager overlaps new-world construction with ongoing training to mask topology-change cost. On dense and MoE models up to 235B parameters, DynaTrain reconfigures a 70B dense model in under 2s and a 235B MoE model in 4.36s, outperforming state-of-the-art checkpoint-based and elastic systems by up to three orders of magnitude while preserving correctness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_18815 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training Wang, Yuanqing Zhang, Yuchen Lin, Hao Hu, Junhao Zhu, Chunyang Zhang, Quanlu Li, Boxun Dai, Guohao Yang, Zhi Cheng, Daning Zhang, Yunquan Wang, Yu Machine Learning Distributed, Parallel, and Cluster Computing Modern large language model (LLM) training is inherently dynamic: resource fluctuations, RLHF phase shifts, and cluster elasticity continually reshape the optimal parallelism layout, posing a significant challenge to existing training frameworks built around a static execution model. We present DynaTrain, a distributed training system for sub-second, online reconfiguration across arbitrary multi-dimensional parallelism. At its core, we propose a Virtual Parameter Space (VPS) abstraction that unifies all distributed training states under one logical coordinate space, turning any parallelism configuration into a deterministic mapping and collapsing complex transition into manageable geometric intersections. On top of VPS, a state routing-and-transition layer executes rank-local transfers under a memory-aware, deadlock-free schedule, and an Elastic Device Manager overlaps new-world construction with ongoing training to mask topology-change cost. On dense and MoE models up to 235B parameters, DynaTrain reconfigures a 70B dense model in under 2s and a 235B MoE model in 4.36s, outperforming state-of-the-art checkpoint-based and elastic systems by up to three orders of magnitude while preserving correctness. |
| title | DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2605.18815 |