An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914621060808704 |
|---|---|
| author | Zhang, Mingjun Hu, Xiaohe Zhang, Menghao Chen, Ziteng Jia, Yanmin Zhang, Yan Liu, Da Chen, Qing Jiao, Fangzheng Chen, Jun Liu, He Zeng, Aohan Duan, Shuaixing Gu, Ruya Jing, Yang Han, Bowen Chen, Wei Xie, Wenqi Hou, Jinlong Cheng, Yuan Zhang, Hongzhou Xu, Bohua Xu, Mingwei Hu, Chunming |
| author_facet | Zhang, Mingjun Hu, Xiaohe Zhang, Menghao Chen, Ziteng Jia, Yanmin Zhang, Yan Liu, Da Chen, Qing Jiao, Fangzheng Chen, Jun Liu, He Zeng, Aohan Duan, Shuaixing Gu, Ruya Jing, Yang Han, Bowen Chen, Wei Xie, Wenqi Hou, Jinlong Cheng, Yuan Zhang, Hongzhou Xu, Bohua Xu, Mingwei Hu, Chunming |
| contents | Large-scale LLM training requires collective communication libraries to exchange data among distributed GPUs. As a company dedicated to building and operating large-scale GPU training clusters, we encounter several practical limitations of NCCL in production, including 1) SM competition between computation and communication, 2) expensive restart costs under link failures, and 3) insufficient observability of transient collective communication anomalies. To address these challenges, we propose VCCL, an efficient, reliable, and observable collective communication library in large-scale GPU training clusters. VCCL removes SM-consuming P2P kernels by moving intra-node data movement and stream dependency enforcement to CPU threads and GPU copy engines. VCCL also introduces a primary-backup QP mechanism to tolerate frequent NIC port failures, and designs a window-based monitor to observe network anomalies at O(μs) level. We opensource VCCL and deploy it in production training clusters for several months. Compared with NCCL, VCCL improves training throughput by up to 5.28% and reduces massive GPU resource wastage through runtime fault tolerance and finegrained monitor. We also share experience and lessons we learned during the deployment of VCCL in large-scale clusters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00991 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters Zhang, Mingjun Hu, Xiaohe Zhang, Menghao Chen, Ziteng Jia, Yanmin Zhang, Yan Liu, Da Chen, Qing Jiao, Fangzheng Chen, Jun Liu, He Zeng, Aohan Duan, Shuaixing Gu, Ruya Jing, Yang Han, Bowen Chen, Wei Xie, Wenqi Hou, Jinlong Cheng, Yuan Zhang, Hongzhou Xu, Bohua Xu, Mingwei Hu, Chunming Distributed, Parallel, and Cluster Computing Large-scale LLM training requires collective communication libraries to exchange data among distributed GPUs. As a company dedicated to building and operating large-scale GPU training clusters, we encounter several practical limitations of NCCL in production, including 1) SM competition between computation and communication, 2) expensive restart costs under link failures, and 3) insufficient observability of transient collective communication anomalies. To address these challenges, we propose VCCL, an efficient, reliable, and observable collective communication library in large-scale GPU training clusters. VCCL removes SM-consuming P2P kernels by moving intra-node data movement and stream dependency enforcement to CPU threads and GPU copy engines. VCCL also introduces a primary-backup QP mechanism to tolerate frequent NIC port failures, and designs a window-based monitor to observe network anomalies at O(μs) level. We opensource VCCL and deploy it in production training clusters for several months. Compared with NCCL, VCCL improves training throughput by up to 5.28% and reduces massive GPU resource wastage through runtime fault tolerance and finegrained monitor. We also share experience and lessons we learned during the deployment of VCCL in large-scale clusters. |
| title | An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2510.00991 |