Scaling Large-scale GNN Training to Thousands of Processors on CPU-based Supercomputers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908379687944192 |
|---|---|
| author | Zhuang, Chen Zhang, Lingqi Wu, Du Chen, Peng Huang, Jiajun Liu, Xin Yokota, Rio Dryden, Nikoli Endo, Toshio Matsuoka, Satoshi Wahib, Mohamed |
| author_facet | Zhuang, Chen Zhang, Lingqi Wu, Du Chen, Peng Huang, Jiajun Liu, Xin Yokota, Rio Dryden, Nikoli Endo, Toshio Matsuoka, Satoshi Wahib, Mohamed |
| contents | Graph Convolutional Networks (GCNs), particularly for large-scale graphs, are crucial across numerous domains. However, training distributed full-batch GCNs on large-scale graphs suffers from inefficient memory access patterns and high communication overhead. To address these challenges, we introduce \method{}, an efficient and scalable distributed GCN training framework tailored for CPU-powered supercomputers. Our contributions are threefold: (1) we develop general and efficient aggregation operators designed for irregular memory access, (2) we propose a hierarchical aggregation scheme that reduces communication costs without altering the graph structure, and (3) we present a communication-aware quantization scheme to enhance performance. Experimental results demonstrate that \method{} achieves a speedup of up to 6$\times$ compared with the SoTA implementations, and scales to 1000s of HPC-grade CPUs on the largest publicly available datasets, without sacrificing model convergence and accuracy. Moreover, due to the effective strong scaling of \method{}, we outperform SoTA GPU-based and CPU-based distributed full-batch GCN training frameworks, in absolute performance, for large-scale graphs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16025 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scaling Large-scale GNN Training to Thousands of Processors on CPU-based Supercomputers Zhuang, Chen Zhang, Lingqi Wu, Du Chen, Peng Huang, Jiajun Liu, Xin Yokota, Rio Dryden, Nikoli Endo, Toshio Matsuoka, Satoshi Wahib, Mohamed Distributed, Parallel, and Cluster Computing Performance Graph Convolutional Networks (GCNs), particularly for large-scale graphs, are crucial across numerous domains. However, training distributed full-batch GCNs on large-scale graphs suffers from inefficient memory access patterns and high communication overhead. To address these challenges, we introduce \method{}, an efficient and scalable distributed GCN training framework tailored for CPU-powered supercomputers. Our contributions are threefold: (1) we develop general and efficient aggregation operators designed for irregular memory access, (2) we propose a hierarchical aggregation scheme that reduces communication costs without altering the graph structure, and (3) we present a communication-aware quantization scheme to enhance performance. Experimental results demonstrate that \method{} achieves a speedup of up to 6$\times$ compared with the SoTA implementations, and scales to 1000s of HPC-grade CPUs on the largest publicly available datasets, without sacrificing model convergence and accuracy. Moreover, due to the effective strong scaling of \method{}, we outperform SoTA GPU-based and CPU-based distributed full-batch GCN training frameworks, in absolute performance, for large-scale graphs. |
| title | Scaling Large-scale GNN Training to Thousands of Processors on CPU-based Supercomputers |
| topic | Distributed, Parallel, and Cluster Computing Performance |
| url | https://arxiv.org/abs/2411.16025 |