Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.02296 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912844720635904 |
|---|---|
| author | Xu, Zhe Hassani, Kaveh Zhang, Si Zeng, Hanqing Yasunaga, Michihiro Wang, Limei Fu, Dongqi Yao, Ning Long, Bo Tong, Hanghang |
| author_facet | Xu, Zhe Hassani, Kaveh Zhang, Si Zeng, Hanqing Yasunaga, Michihiro Wang, Limei Fu, Dongqi Yao, Ning Long, Bo Tong, Hanghang |
| contents | Language Models (LMs) are increasingly challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs), in graph learning tasks. Following this trend, we propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks, without requiring any architectural modification. By preserving the LM's original architecture, our approach retains a key benefit of LM instruction tuning: the ability to jointly train on diverse datasets, fostering greater flexibility and efficiency. To achieve this, we introduce two key augmentation strategies: (1) Enriching LMs' input using topological and semantic retrieval methods, which provide richer contextual information, and (2) guiding the LMs' classification process through a lightweight GNN classifier that effectively prunes class candidates. Our experiments on real-world datasets show that backbone Flan-T5 LMs equipped with these augmentation strategies outperform SOTA text-output node classifiers and are comparable to top-performing vector-output node classifiers. By bridging the gap between specialized node classifiers and general LMs, this work paves the way for more versatile and widely applicable graph learning models. We will open-source the code upon publication. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02296 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How to Make LMs Strong Node Classifiers? Xu, Zhe Hassani, Kaveh Zhang, Si Zeng, Hanqing Yasunaga, Michihiro Wang, Limei Fu, Dongqi Yao, Ning Long, Bo Tong, Hanghang Computation and Language Language Models (LMs) are increasingly challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs), in graph learning tasks. Following this trend, we propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks, without requiring any architectural modification. By preserving the LM's original architecture, our approach retains a key benefit of LM instruction tuning: the ability to jointly train on diverse datasets, fostering greater flexibility and efficiency. To achieve this, we introduce two key augmentation strategies: (1) Enriching LMs' input using topological and semantic retrieval methods, which provide richer contextual information, and (2) guiding the LMs' classification process through a lightweight GNN classifier that effectively prunes class candidates. Our experiments on real-world datasets show that backbone Flan-T5 LMs equipped with these augmentation strategies outperform SOTA text-output node classifiers and are comparable to top-performing vector-output node classifiers. By bridging the gap between specialized node classifiers and general LMs, this work paves the way for more versatile and widely applicable graph learning models. We will open-source the code upon publication. |
| title | How to Make LMs Strong Node Classifiers? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.02296 |