Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910046665834496 |
|---|---|
| author | Borde, Haitz Sáez de Ocáriz Lukoianov, Artem Kratsios, Anastasis Bronstein, Michael Dong, Xiaowen |
| author_facet | Borde, Haitz Sáez de Ocáriz Lukoianov, Artem Kratsios, Anastasis Bronstein, Michael Dong, Xiaowen |
| contents | We propose Scalable Message Passing Neural Networks (SMPNNs) and demonstrate that, by integrating standard convolutional message passing into a Pre-Layer Normalization Transformer-style block instead of attention, we can produce high-performing deep message-passing-based Graph Neural Networks (GNNs). This modification yields results competitive with the state-of-the-art in large graph transductive learning, particularly outperforming the best Graph Transformers in the literature, without requiring the otherwise computationally and memory-expensive attention mechanism. Our architecture not only scales to large graphs but also makes it possible to construct deep message-passing networks, unlike simple GNNs, which have traditionally been constrained to shallow architectures due to oversmoothing. Moreover, we provide a new theoretical analysis of oversmoothing based on universal approximation which we use to motivate SMPNNs. We show that in the context of graph convolutions, residual connections are necessary for maintaining the universal approximation properties of downstream learners and that removing them can lead to a loss of universality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_00835 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learning Borde, Haitz Sáez de Ocáriz Lukoianov, Artem Kratsios, Anastasis Bronstein, Michael Dong, Xiaowen Machine Learning We propose Scalable Message Passing Neural Networks (SMPNNs) and demonstrate that, by integrating standard convolutional message passing into a Pre-Layer Normalization Transformer-style block instead of attention, we can produce high-performing deep message-passing-based Graph Neural Networks (GNNs). This modification yields results competitive with the state-of-the-art in large graph transductive learning, particularly outperforming the best Graph Transformers in the literature, without requiring the otherwise computationally and memory-expensive attention mechanism. Our architecture not only scales to large graphs but also makes it possible to construct deep message-passing networks, unlike simple GNNs, which have traditionally been constrained to shallow architectures due to oversmoothing. Moreover, we provide a new theoretical analysis of oversmoothing based on universal approximation which we use to motivate SMPNNs. We show that in the context of graph convolutions, residual connections are necessary for maintaining the universal approximation properties of downstream learners and that removing them can lead to a loss of universality. |
| title | Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.00835 |