Don't Let a Few Network Failures Slow the Entire AllReduce
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914621793763328 |
|---|---|
| author | Chen, Peiqing Jiang, Jiedong Yu, Nengneng Wang, Yuefeng Xiong, Sixian Wang, Wei Liu, Zaoxing |
| author_facet | Chen, Peiqing Jiang, Jiedong Yu, Nengneng Wang, Yuefeng Xiong, Sixian Wang, Wei Liu, Zaoxing |
| contents | Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication libraries such as NCCL mitigate network failures by rerouting traffic through surviving NICs on the same server, trading reduced inter-node bandwidth for uninterrupted training. However, the degraded server remains on the critical path of the standard ring algorithm, slowing the entire collective. We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth and show that when the straggler retains at least half of its original bandwidth, the unavoidable overhead relative to the fault-free optimum is only O(1/p) for p GPUs. We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound. Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes: under practical network failures with up to 50% bandwidth loss, OptCC completes AllReduce within 2-6% of NCCL's fault-free ring performance, whereas the state-of-the-art incurs up to 57% overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01680 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Don't Let a Few Network Failures Slow the Entire AllReduce Chen, Peiqing Jiang, Jiedong Yu, Nengneng Wang, Yuefeng Xiong, Sixian Wang, Wei Liu, Zaoxing Distributed, Parallel, and Cluster Computing Machine Learning Networking and Internet Architecture Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication libraries such as NCCL mitigate network failures by rerouting traffic through surviving NICs on the same server, trading reduced inter-node bandwidth for uninterrupted training. However, the degraded server remains on the critical path of the standard ring algorithm, slowing the entire collective. We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth and show that when the straggler retains at least half of its original bandwidth, the unavoidable overhead relative to the fault-free optimum is only O(1/p) for p GPUs. We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound. Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes: under practical network failures with up to 50% bandwidth loss, OptCC completes AllReduce within 2-6% of NCCL's fault-free ring performance, whereas the state-of-the-art incurs up to 57% overhead. |
| title | Don't Let a Few Network Failures Slow the Entire AllReduce |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning Networking and Internet Architecture |
| url | https://arxiv.org/abs/2606.01680 |