MSCCL++: Rethinking GPU Communication Abstractions for AI Inference
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910001379934208 |
|---|---|
| author | Hwang, Changho Cheng, Peng Dathathri, Roshan Jangda, Abhinav Maleki, Saeed Musuvathi, Madan Saarikivi, Olli Shah, Aashaka Yang, Ziyue Li, Binyang Rocha, Caio Zhou, Qinghua Ghazimirsaeed, Mahdieh Anantharamu, Sreevatsa Jose, Jithin |
| author_facet | Hwang, Changho Cheng, Peng Dathathri, Roshan Jangda, Abhinav Maleki, Saeed Musuvathi, Madan Saarikivi, Olli Shah, Aashaka Yang, Ziyue Li, Binyang Rocha, Caio Zhou, Qinghua Ghazimirsaeed, Mahdieh Anantharamu, Sreevatsa Jose, Jithin |
| contents | AI applications increasingly run on fast-evolving, heterogeneous hardware to maximize performance, but general-purpose libraries lag in supporting these features. Performance-minded programmers often build custom communication stacks that are fast but error-prone and non-portable.
This paper introduces MSCCL++, a design methodology for developing high-performance, portable communication kernels. It provides (1) a low-level, performance-preserving primitive interface that exposes minimal hardware abstractions while hiding the complexities of synchronization and consistency, (2) a higher-level DSL for application developers to implement workload-specific communication algorithms, and (3) a library of efficient algorithms implementing the standard collective API, enabling adoption by users with minimal expertise.
Compared to state-of-the-art baselines, MSCCL++ achieves geomean speedups of $1.7\times$ (up to $5.4\times$) for collective communication and $1.2\times$ (up to $1.38\times$) for AI inference workloads. MSCCL++ is in production of multiple AI services provided by Microsoft Azure, and has also been adopted by RCCL, the GPU collective communication library maintained by AMD. MSCCL++ is open source and available at https://github.com/microsoft/mscclpp . Our two years of experience with MSCCL++ suggests that its abstractions are robust, enabling support for new hardware features, such as multimem, within weeks of development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09014 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MSCCL++: Rethinking GPU Communication Abstractions for AI Inference Hwang, Changho Cheng, Peng Dathathri, Roshan Jangda, Abhinav Maleki, Saeed Musuvathi, Madan Saarikivi, Olli Shah, Aashaka Yang, Ziyue Li, Binyang Rocha, Caio Zhou, Qinghua Ghazimirsaeed, Mahdieh Anantharamu, Sreevatsa Jose, Jithin Distributed, Parallel, and Cluster Computing Artificial Intelligence AI applications increasingly run on fast-evolving, heterogeneous hardware to maximize performance, but general-purpose libraries lag in supporting these features. Performance-minded programmers often build custom communication stacks that are fast but error-prone and non-portable. This paper introduces MSCCL++, a design methodology for developing high-performance, portable communication kernels. It provides (1) a low-level, performance-preserving primitive interface that exposes minimal hardware abstractions while hiding the complexities of synchronization and consistency, (2) a higher-level DSL for application developers to implement workload-specific communication algorithms, and (3) a library of efficient algorithms implementing the standard collective API, enabling adoption by users with minimal expertise. Compared to state-of-the-art baselines, MSCCL++ achieves geomean speedups of $1.7\times$ (up to $5.4\times$) for collective communication and $1.2\times$ (up to $1.38\times$) for AI inference workloads. MSCCL++ is in production of multiple AI services provided by Microsoft Azure, and has also been adopted by RCCL, the GPU collective communication library maintained by AMD. MSCCL++ is open source and available at https://github.com/microsoft/mscclpp . Our two years of experience with MSCCL++ suggests that its abstractions are robust, enabling support for new hardware features, such as multimem, within weeks of development. |
| title | MSCCL++: Rethinking GPU Communication Abstractions for AI Inference |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2504.09014 |