NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908932446879744 |
|---|---|
| author | Goldman, Amos Boker, Nimrod Sheraizin, Maayan Admoni, Nimrod Polyakov, Artem Bhattacharya, Subhadeep Yu, Fan Sun, Kai Theodorakis, Georgios Yin, Hsin-Chun Gootzen, Peter-Jan Shafi, Aamir Ravid, Assaf Di Girolamo, Salvatore Dinan, James Li, Xiaofan Venkata, Manjunath Gorentla Bloch, Gil |
| author_facet | Goldman, Amos Boker, Nimrod Sheraizin, Maayan Admoni, Nimrod Polyakov, Artem Bhattacharya, Subhadeep Yu, Fan Sun, Kai Theodorakis, Georgios Yin, Hsin-Chun Gootzen, Peter-Jan Shafi, Aamir Ravid, Assaf Di Girolamo, Salvatore Dinan, James Li, Xiaofan Venkata, Manjunath Gorentla Bloch, Gil |
| contents | Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE dispatch and combine operations.
This paper presents NCCL EP (Expert Parallelism), a ground-up MoE communication library built entirely on NCCL's Device API. NCCL EP provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces, supporting Low-Latency (LL) mode for inference decoding and High-Throughput (HT) mode for training and inference prefill. LL targets small batch sizes (1-128 tokens) using direct all-to-all RDMA+NVLink mesh connectivity with double-buffered communication for overlapping dispatch and combine phases. HT targets large batches (4096+ tokens) using hierarchical communication that aggregates tokens within NVLink domains before inter-node RDMA transmission. Both modes leverage Device API for both intra- and inter-node communications, taking advantage of its topology awareness and optimized GPU-initiated implementation.
We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration. By building MoE communication natively within NCCL, NCCL EP provides a supported path for expert parallelism on current and emerging NVIDIA platforms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_13606 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL Goldman, Amos Boker, Nimrod Sheraizin, Maayan Admoni, Nimrod Polyakov, Artem Bhattacharya, Subhadeep Yu, Fan Sun, Kai Theodorakis, Georgios Yin, Hsin-Chun Gootzen, Peter-Jan Shafi, Aamir Ravid, Assaf Di Girolamo, Salvatore Dinan, James Li, Xiaofan Venkata, Manjunath Gorentla Bloch, Gil Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture Machine Learning 68W10, 68M10, 65Y05 C.2.1; C.2.4; C.1.2; C.1.4 Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE dispatch and combine operations. This paper presents NCCL EP (Expert Parallelism), a ground-up MoE communication library built entirely on NCCL's Device API. NCCL EP provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces, supporting Low-Latency (LL) mode for inference decoding and High-Throughput (HT) mode for training and inference prefill. LL targets small batch sizes (1-128 tokens) using direct all-to-all RDMA+NVLink mesh connectivity with double-buffered communication for overlapping dispatch and combine phases. HT targets large batches (4096+ tokens) using hierarchical communication that aggregates tokens within NVLink domains before inter-node RDMA transmission. Both modes leverage Device API for both intra- and inter-node communications, taking advantage of its topology awareness and optimized GPU-initiated implementation. We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration. By building MoE communication natively within NCCL, NCCL EP provides a supported path for expert parallelism on current and emerging NVIDIA platforms. |
| title | NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture Machine Learning 68W10, 68M10, 65Y05 C.2.1; C.2.4; C.1.2; C.1.4 |
| url | https://arxiv.org/abs/2603.13606 |