HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911731542917120 |
|---|---|
| author | Wang, Yuejie Chang, Tao Zhao, Yuanyuan Ao, Yulong Gu, Zeyu Li, Zhiyu Jia, Yanmin Zhang, Yan Zhang, Mingjun Liu, He He, Yongzhe Lin, Yonghua Liu, Guyue |
| author_facet | Wang, Yuejie Chang, Tao Zhao, Yuanyuan Ao, Yulong Gu, Zeyu Li, Zhiyu Jia, Yanmin Zhang, Yan Zhang, Mingjun Liu, He He, Yongzhe Lin, Yonghua Liu, Guyue |
| contents | Training Large Language Models (LLMs) on heterogeneous clusters presents significant challenges for collective communication, as hardware from multiple vendors introduces diverse network and computational characteristics. Existing collective communication frameworks (e.g., NCCL, RCCL) designed for homogeneous environments fail to address mixed-hardware setups, while communication libraries with heterogeneous support (e.g., Gloo, OpenMPI) incur heavy overhead in the data path.
This paper presents HetCCL, a framework that enables heterogeneous collective communication by efficient P2P transport across heterogeneous devices (e.g., GPUs), eliminating the host-device memory copy overhead while offloading the control to the CPUs. For combining collectives (e.g., AllReduce, ReduceScatter), HetCCL introduces a border-communicator mechanism that achieves vendor independence by using the intrinsic reduction in the combining collectives in vendor collective communication libraries. With efficient heterogeneous P2P transport and portable reduction mechanism, HetCCL proposes a hierarchical topology abstraction for heterogeneous clusters, dissecting collective communication into cluster-level primitives that guarantee optimal cross-cluster data transfer volume and optimal bandwidth utilization.
We implement HetCCL with 4 different vendor support and evaluate it in 4 heterogeneous settings with benchmarks and end-to-end LLM tasks. Our evaluation shows that HetCCL achieves 17-19x higher bandwidth than Gloo in heterogeneous communications, and speeds up end-to-end training by up to 16.9% in the per-step-time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_31000 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters Wang, Yuejie Chang, Tao Zhao, Yuanyuan Ao, Yulong Gu, Zeyu Li, Zhiyu Jia, Yanmin Zhang, Yan Zhang, Mingjun Liu, He He, Yongzhe Lin, Yonghua Liu, Guyue Networking and Internet Architecture Machine Learning Training Large Language Models (LLMs) on heterogeneous clusters presents significant challenges for collective communication, as hardware from multiple vendors introduces diverse network and computational characteristics. Existing collective communication frameworks (e.g., NCCL, RCCL) designed for homogeneous environments fail to address mixed-hardware setups, while communication libraries with heterogeneous support (e.g., Gloo, OpenMPI) incur heavy overhead in the data path. This paper presents HetCCL, a framework that enables heterogeneous collective communication by efficient P2P transport across heterogeneous devices (e.g., GPUs), eliminating the host-device memory copy overhead while offloading the control to the CPUs. For combining collectives (e.g., AllReduce, ReduceScatter), HetCCL introduces a border-communicator mechanism that achieves vendor independence by using the intrinsic reduction in the combining collectives in vendor collective communication libraries. With efficient heterogeneous P2P transport and portable reduction mechanism, HetCCL proposes a hierarchical topology abstraction for heterogeneous clusters, dissecting collective communication into cluster-level primitives that guarantee optimal cross-cluster data transfer volume and optimal bandwidth utilization. We implement HetCCL with 4 different vendor support and evaluate it in 4 heterogeneous settings with benchmarks and end-to-end LLM tasks. Our evaluation shows that HetCCL achieves 17-19x higher bandwidth than Gloo in heterogeneous communications, and speeds up end-to-end training by up to 16.9% in the per-step-time. |
| title | HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters |
| topic | Networking and Internet Architecture Machine Learning |
| url | https://arxiv.org/abs/2605.31000 |