Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918140270608384 |
|---|---|
| author | Feng, Wenjiao Xiao, Rongxing Li, Zonghang Yu, Hongfang Sun, Gang Luo, Long Guizani, Mohsen Ho, Qirong Liu, Steve |
| author_facet | Feng, Wenjiao Xiao, Rongxing Li, Zonghang Yu, Hongfang Sun, Gang Luo, Long Guizani, Mohsen Ho, Qirong Liu, Steve |
| contents | Node and link churn in multi-party, cross-region clusters over wide-area networks (WANs) often disrupts distributed training. However, checkpoint-based recovery and cloud-centric autoscaling react slowly and assume centralized control, which is misaligned with the self-governed setup where institutions can freely join and leave. This paper proposes Chaos, a multi-party distributed training system with self-healing and autoscaling, enabling robust and elastic training under churn. It speeds up autoscaling via multi-neighbor state replication and model sharding. We formalize the sharding and assignment as a MINLP that captures WAN heterogeneity, and reduce it to a tractable MILP by analyzing its monotonicity on a divisibility chain. By establishing an equivalence, we derive a greedy algorithm that follows optimality rules and yields the optimal solution in polynomial time. Chaos uses a cluster monitor to track resource and topology changes, and handles scaling events through peer negotiation protocols, enabling fully self-governed autoscaling among institutions. Experiments show that Chaos has substantially lower scale-out delay than Pollux, Elan, and Autoscaling, and handles scale-in, connect-link, and disconnect-link events within 20ms. It also delivers the lowest idle time, showing superior resource use and scalability as the cluster grows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12815 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training Feng, Wenjiao Xiao, Rongxing Li, Zonghang Yu, Hongfang Sun, Gang Luo, Long Guizani, Mohsen Ho, Qirong Liu, Steve Distributed, Parallel, and Cluster Computing Artificial Intelligence 68T99 I.2.11 Node and link churn in multi-party, cross-region clusters over wide-area networks (WANs) often disrupts distributed training. However, checkpoint-based recovery and cloud-centric autoscaling react slowly and assume centralized control, which is misaligned with the self-governed setup where institutions can freely join and leave. This paper proposes Chaos, a multi-party distributed training system with self-healing and autoscaling, enabling robust and elastic training under churn. It speeds up autoscaling via multi-neighbor state replication and model sharding. We formalize the sharding and assignment as a MINLP that captures WAN heterogeneity, and reduce it to a tractable MILP by analyzing its monotonicity on a divisibility chain. By establishing an equivalence, we derive a greedy algorithm that follows optimality rules and yields the optimal solution in polynomial time. Chaos uses a cluster monitor to track resource and topology changes, and handles scaling events through peer negotiation protocols, enabling fully self-governed autoscaling among institutions. Experiments show that Chaos has substantially lower scale-out delay than Pollux, Elan, and Autoscaling, and handles scale-in, connect-link, and disconnect-link events within 20ms. It also delivers the lowest idle time, showing superior resource use and scalability as the cluster grows. |
| title | Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence 68T99 I.2.11 |
| url | https://arxiv.org/abs/2505.12815 |