MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916465669570560 |
|---|---|
| author | Sarkar, Aishwarya Ghosh, Sayan Tallent, Nathan R. Jannesari, Ali |
| author_facet | Sarkar, Aishwarya Ghosh, Sayan Tallent, Nathan R. Jannesari, Ali |
| contents | Graph Neural Networks (GNN) are indispensable in learning from graph-structured data, yet their rising computational costs, especially on massively connected graphs, pose significant challenges in terms of execution performance. To tackle this, distributed-memory solutions such as partitioning the graph to concurrently train multiple replicas of GNNs are in practice. However, approaches requiring a partitioned graph usually suffer from communication overhead and load imbalance, even under optimal partitioning and communication strategies due to irregularities in the neighborhood minibatch sampling.
This paper proposes practical trade-offs for improving the sampling and communication overheads for representation learning on distributed graphs (using popular GraphSAGE architecture) by developing a parameterized continuous prefetch and eviction scheme on top of the state-of-the-art Amazon DistDGL distributed GNN framework, demonstrating about 15-40% improvement in end-to-end training performance on the National Energy Research Scientific Computing Center's (NERSC) Perlmutter supercomputer for various OGB datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_22697 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs Sarkar, Aishwarya Ghosh, Sayan Tallent, Nathan R. Jannesari, Ali Distributed, Parallel, and Cluster Computing Machine Learning Performance Graph Neural Networks (GNN) are indispensable in learning from graph-structured data, yet their rising computational costs, especially on massively connected graphs, pose significant challenges in terms of execution performance. To tackle this, distributed-memory solutions such as partitioning the graph to concurrently train multiple replicas of GNNs are in practice. However, approaches requiring a partitioned graph usually suffer from communication overhead and load imbalance, even under optimal partitioning and communication strategies due to irregularities in the neighborhood minibatch sampling. This paper proposes practical trade-offs for improving the sampling and communication overheads for representation learning on distributed graphs (using popular GraphSAGE architecture) by developing a parameterized continuous prefetch and eviction scheme on top of the state-of-the-art Amazon DistDGL distributed GNN framework, demonstrating about 15-40% improvement in end-to-end training performance on the National Energy Research Scientific Computing Center's (NERSC) Perlmutter supercomputer for various OGB datasets. |
| title | MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning Performance |
| url | https://arxiv.org/abs/2410.22697 |