Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.19986 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917939220840448 |
|---|---|
| author | Vatter, Jana Zayats, Mykhaylo Galindo, Marcos Martínez López, Vanessa Mayer, Ruben Jacobsen, Hans-Arno Lam, Hoang Thanh |
| author_facet | Vatter, Jana Zayats, Mykhaylo Galindo, Marcos Martínez López, Vanessa Mayer, Ruben Jacobsen, Hans-Arno Lam, Hoang Thanh |
| contents | With the ever-growing size of real-world graphs, numerous techniques to overcome resource limitations when training Graph Neural Networks (GNNs) have been developed. One such approach, GNNAutoScale (GAS), uses graph partitioning to enable training under constrained GPU memory. GAS also stores historical embedding vectors, which are retrieved from one-hop neighbors in other partitions, ensuring critical information is captured across partition boundaries. The historical embeddings which come from the previous training iteration are stale compared to the GAS estimated embeddings, resulting in approximation errors of the training algorithm. Furthermore, these errors accumulate over multiple layers, leading to suboptimal node embeddings. To address this shortcoming, we propose two enhancements: first, WaveGAS, inspired by waveform relaxation, performs multiple forward passes within GAS before the backward pass, refining the approximation of historical embeddings and gradients to improve accuracy; second, a gradient-tracking method that stores and utilizes more accurate historical gradients during training. Empirical results show that WaveGAS enhances GAS and achieves better accuracy, even outperforming methods that train on full graphs, thanks to its robust estimation of node embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19986 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | WaveGAS: Waveform Relaxation for Scaling Graph Neural Networks Vatter, Jana Zayats, Mykhaylo Galindo, Marcos Martínez López, Vanessa Mayer, Ruben Jacobsen, Hans-Arno Lam, Hoang Thanh Machine Learning With the ever-growing size of real-world graphs, numerous techniques to overcome resource limitations when training Graph Neural Networks (GNNs) have been developed. One such approach, GNNAutoScale (GAS), uses graph partitioning to enable training under constrained GPU memory. GAS also stores historical embedding vectors, which are retrieved from one-hop neighbors in other partitions, ensuring critical information is captured across partition boundaries. The historical embeddings which come from the previous training iteration are stale compared to the GAS estimated embeddings, resulting in approximation errors of the training algorithm. Furthermore, these errors accumulate over multiple layers, leading to suboptimal node embeddings. To address this shortcoming, we propose two enhancements: first, WaveGAS, inspired by waveform relaxation, performs multiple forward passes within GAS before the backward pass, refining the approximation of historical embeddings and gradients to improve accuracy; second, a gradient-tracking method that stores and utilizes more accurate historical gradients during training. Empirical results show that WaveGAS enhances GAS and achieves better accuracy, even outperforming methods that train on full graphs, thanks to its robust estimation of node embeddings. |
| title | WaveGAS: Waveform Relaxation for Scaling Graph Neural Networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.19986 |