ParaFormer: A Generalized PageRank Graph Transformer for Graph Representation Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912768924319744 |
|---|---|
| author | Yuan, Chaohao Song, Zhenjie Kuruoglu, Ercan Engin Zhao, Kangfei Liu, Yang Zhao, Deli Cheng, Hong Rong, Yu |
| author_facet | Yuan, Chaohao Song, Zhenjie Kuruoglu, Ercan Engin Zhao, Kangfei Liu, Yang Zhao, Deli Cheng, Hong Rong, Yu |
| contents | Graph Transformers (GTs) have emerged as a promising graph learning tool, leveraging their all-pair connected property to effectively capture global information. To address the over-smoothing problem in deep GNNs, global attention was initially introduced, eliminating the necessity for using deep GNNs. However, through empirical and theoretical analysis, we verify that the introduced global attention exhibits severe over-smoothing, causing node representations to become indistinguishable due to its inherent low-pass filtering. This effect is even stronger than that observed in GNNs. To mitigate this, we propose PageRank Transformer (ParaFormer), which features a PageRank-enhanced attention module designed to mimic the behavior of deep Transformers. We theoretically and empirically demonstrate that ParaFormer mitigates over-smoothing by functioning as an adaptive-pass filter. Experiments show that ParaFormer achieves consistent performance improvements across both node classification and graph classification tasks on 11 datasets ranging from thousands to millions of nodes, validating its efficacy. The supplementary material, including code and appendix, can be found in https://github.com/chaohaoyuan/ParaFormer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_14619 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ParaFormer: A Generalized PageRank Graph Transformer for Graph Representation Learning Yuan, Chaohao Song, Zhenjie Kuruoglu, Ercan Engin Zhao, Kangfei Liu, Yang Zhao, Deli Cheng, Hong Rong, Yu Machine Learning Graph Transformers (GTs) have emerged as a promising graph learning tool, leveraging their all-pair connected property to effectively capture global information. To address the over-smoothing problem in deep GNNs, global attention was initially introduced, eliminating the necessity for using deep GNNs. However, through empirical and theoretical analysis, we verify that the introduced global attention exhibits severe over-smoothing, causing node representations to become indistinguishable due to its inherent low-pass filtering. This effect is even stronger than that observed in GNNs. To mitigate this, we propose PageRank Transformer (ParaFormer), which features a PageRank-enhanced attention module designed to mimic the behavior of deep Transformers. We theoretically and empirically demonstrate that ParaFormer mitigates over-smoothing by functioning as an adaptive-pass filter. Experiments show that ParaFormer achieves consistent performance improvements across both node classification and graph classification tasks on 11 datasets ranging from thousands to millions of nodes, validating its efficacy. The supplementary material, including code and appendix, can be found in https://github.com/chaohaoyuan/ParaFormer. |
| title | ParaFormer: A Generalized PageRank Graph Transformer for Graph Representation Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.14619 |