Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929262502608896 |
|---|---|
| author | Li, Junxian Shi, Bin Cui, Erfei Wei, Hua Zheng, Qinghua |
| author_facet | Li, Junxian Shi, Bin Cui, Erfei Wei, Hua Zheng, Qinghua |
| contents | We study the challenging problem for inference tasks on large-scale graph datasets of Graph Neural Networks: huge time and memory consumption, and try to overcome it by reducing reliance on graph structure. Even though distilling graph knowledge to student MLP is an excellent idea, it faces two major problems of positional information loss and low generalization. To solve the problems, we propose a new three-stage multitask distillation framework. In detail, we use Positional Encoding to capture positional information. Also, we introduce Neural Heat Kernels responsible for graph data processing in GNN and utilize hidden layer outputs matching for better performance of student MLP's hidden layers. To the best of our knowledge, it is the first work to include hidden layer distillation for student MLP on graphs and to combine graph Positional Encoding with MLP. We test its performance and robustness with several settings and draw the conclusion that our work can outperform well with good stability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01079 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework Li, Junxian Shi, Bin Cui, Erfei Wei, Hua Zheng, Qinghua Machine Learning Artificial Intelligence We study the challenging problem for inference tasks on large-scale graph datasets of Graph Neural Networks: huge time and memory consumption, and try to overcome it by reducing reliance on graph structure. Even though distilling graph knowledge to student MLP is an excellent idea, it faces two major problems of positional information loss and low generalization. To solve the problems, we propose a new three-stage multitask distillation framework. In detail, we use Positional Encoding to capture positional information. Also, we introduce Neural Heat Kernels responsible for graph data processing in GNN and utilize hidden layer outputs matching for better performance of student MLP's hidden layers. To the best of our knowledge, it is the first work to include hidden layer distillation for student MLP on graphs and to combine graph Positional Encoding with MLP. We test its performance and robustness with several settings and draw the conclusion that our work can outperform well with good stability. |
| title | Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.01079 |