GPU-Initiated Networking for NCCL
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909922258583552 |
|---|---|
| author | Hamidouche, Khaled Bachan, John Markthub, Pak Gootzen, Peter-Jan Agostini, Elena Jeaugey, Sylvain Shafi, Aamir Theodorakis, Georgios Venkata, Manjunath Gorentla |
| author_facet | Hamidouche, Khaled Bachan, John Markthub, Pak Gootzen, Peter-Jan Agostini, Elena Jeaugey, Sylvain Shafi, Aamir Theodorakis, Georgios Venkata, Manjunath Gorentla |
| contents | Modern AI workloads, especially Mixture-of-Experts (MoE) architectures, increasingly demand low-latency, fine-grained GPU-to-GPU communication with device-side control. Traditional GPU communication follows a host-initiated model, where the CPU orchestrates all communication operations - a characteristic of the CUDA runtime. Although robust for collective operations, applications requiring tight integration of computation and communication can benefit from device-initiated communication that eliminates CPU coordination overhead.
NCCL 2.28 introduces the Device API with three operation modes: Load/Store Accessible (LSA) for NVLink/PCIe, Multimem for NVLink SHARP, and GPU-Initiated Networking (GIN) for network RDMA. This paper presents the GIN architecture, design, semantics, and highlights its impact on MoE communication. GIN builds on a three-layer architecture: i) NCCL Core host-side APIs for device communicator setup and collective memory window registration; ii) Device-side APIs for remote memory operations callable from CUDA kernels; and iii) A network plugin architecture with dual semantics (GPUDirect Async Kernel-Initiated and Proxy) for broad hardware support. The GPUDirect Async Kernel-Initiated backend leverages DOCA GPUNetIO for direct GPU-to-NIC communication, while the Proxy backend provides equivalent functionality via lock-free GPU-to-CPU queues over standard RDMA networks. We demonstrate GIN's practicality through integration with DeepEP, an MoE communication library. Comprehensive benchmarking shows that GIN provides device-initiated communication within NCCL's unified runtime, combining low-latency operations with NCCL's collective algorithms and production infrastructure. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_15076 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GPU-Initiated Networking for NCCL Hamidouche, Khaled Bachan, John Markthub, Pak Gootzen, Peter-Jan Agostini, Elena Jeaugey, Sylvain Shafi, Aamir Theodorakis, Georgios Venkata, Manjunath Gorentla Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture Machine Learning 68W10 (Primary), 68M10, 65Y05 (Secondary) C.2.1; C.2.4; C.1.2; C.1.4 Modern AI workloads, especially Mixture-of-Experts (MoE) architectures, increasingly demand low-latency, fine-grained GPU-to-GPU communication with device-side control. Traditional GPU communication follows a host-initiated model, where the CPU orchestrates all communication operations - a characteristic of the CUDA runtime. Although robust for collective operations, applications requiring tight integration of computation and communication can benefit from device-initiated communication that eliminates CPU coordination overhead. NCCL 2.28 introduces the Device API with three operation modes: Load/Store Accessible (LSA) for NVLink/PCIe, Multimem for NVLink SHARP, and GPU-Initiated Networking (GIN) for network RDMA. This paper presents the GIN architecture, design, semantics, and highlights its impact on MoE communication. GIN builds on a three-layer architecture: i) NCCL Core host-side APIs for device communicator setup and collective memory window registration; ii) Device-side APIs for remote memory operations callable from CUDA kernels; and iii) A network plugin architecture with dual semantics (GPUDirect Async Kernel-Initiated and Proxy) for broad hardware support. The GPUDirect Async Kernel-Initiated backend leverages DOCA GPUNetIO for direct GPU-to-NIC communication, while the Proxy backend provides equivalent functionality via lock-free GPU-to-CPU queues over standard RDMA networks. We demonstrate GIN's practicality through integration with DeepEP, an MoE communication library. Comprehensive benchmarking shows that GIN provides device-initiated communication within NCCL's unified runtime, combining low-latency operations with NCCL's collective algorithms and production infrastructure. |
| title | GPU-Initiated Networking for NCCL |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture Machine Learning 68W10 (Primary), 68M10, 65Y05 (Secondary) C.2.1; C.2.4; C.1.2; C.1.4 |
| url | https://arxiv.org/abs/2511.15076 |