SkipNode: On Alleviating Performance Degradation for Deep Graph Convolutional Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916102128271360 |
|---|---|
| author | Lu, Weigang Zhan, Yibing Lin, Binbin Guan, Ziyu Liu, Liu Yu, Baosheng Zhao, Wei Yang, Yaming Tao, Dacheng |
| author_facet | Lu, Weigang Zhan, Yibing Lin, Binbin Guan, Ziyu Liu, Liu Yu, Baosheng Zhao, Wei Yang, Yaming Tao, Dacheng |
| contents | Graph Convolutional Networks (GCNs) suffer from performance degradation when models go deeper. However, earlier works only attributed the performance degeneration to over-smoothing. In this paper, we conduct theoretical and experimental analysis to explore the fundamental causes of performance degradation in deep GCNs: over-smoothing and gradient vanishing have a mutually reinforcing effect that causes the performance to deteriorate more quickly in deep GCNs. On the other hand, existing anti-over-smoothing methods all perform full convolutions up to the model depth. They could not well resist the exponential convergence of over-smoothing due to model depth increasing. In this work, we propose a simple yet effective plug-and-play module, Skipnode, to overcome the performance degradation of deep GCNs. It samples graph nodes in each convolutional layer to skip the convolution operation. In this way, both over-smoothing and gradient vanishing can be effectively suppressed since (1) not all nodes'features propagate through full layers and, (2) the gradient can be directly passed back through ``skipped'' nodes. We provide both theoretical analysis and empirical evaluation to demonstrate the efficacy of Skipnode and its superiority over SOTA baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2112_11628 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | SkipNode: On Alleviating Performance Degradation for Deep Graph Convolutional Networks Lu, Weigang Zhan, Yibing Lin, Binbin Guan, Ziyu Liu, Liu Yu, Baosheng Zhao, Wei Yang, Yaming Tao, Dacheng Machine Learning Graph Convolutional Networks (GCNs) suffer from performance degradation when models go deeper. However, earlier works only attributed the performance degeneration to over-smoothing. In this paper, we conduct theoretical and experimental analysis to explore the fundamental causes of performance degradation in deep GCNs: over-smoothing and gradient vanishing have a mutually reinforcing effect that causes the performance to deteriorate more quickly in deep GCNs. On the other hand, existing anti-over-smoothing methods all perform full convolutions up to the model depth. They could not well resist the exponential convergence of over-smoothing due to model depth increasing. In this work, we propose a simple yet effective plug-and-play module, Skipnode, to overcome the performance degradation of deep GCNs. It samples graph nodes in each convolutional layer to skip the convolution operation. In this way, both over-smoothing and gradient vanishing can be effectively suppressed since (1) not all nodes'features propagate through full layers and, (2) the gradient can be directly passed back through ``skipped'' nodes. We provide both theoretical analysis and empirical evaluation to demonstrate the efficacy of Skipnode and its superiority over SOTA baselines. |
| title | SkipNode: On Alleviating Performance Degradation for Deep Graph Convolutional Networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2112.11628 |