Improving global awareness of linkset predictions using Cross-Attentive Modulation tokens
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912035909926912 |
|---|---|
| author | Marcoccia, Félix Adjih, Cédric Mühlethaler, Paul |
| author_facet | Marcoccia, Félix Adjih, Cédric Mühlethaler, Paul |
| contents | This work introduces Cross-Attentive Modulation (CAM) tokens, which are tokens whose initial value is learned, gather information through cross-attention, and modulate the nodes and edges accordingly. These tokens are meant to improve the global awareness of link predictions models which, based on graph neural networks, can struggle to capture graph-level features. This lack of ability to feature high level representations is particularly limiting when predicting multiple or entire sets of links. We implement CAM tokens in a simple attention-based link prediction model and in a graph transformer, which we also use in a denoising diffusion framework. A brief introduction to our toy datasets will then be followed by benchmarks which prove that CAM token improve the performance of the model they supplement and outperform a baseline with diverse statistical graph attributes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_19375 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving global awareness of linkset predictions using Cross-Attentive Modulation tokens Marcoccia, Félix Adjih, Cédric Mühlethaler, Paul Social and Information Networks Machine Learning I.2.6 This work introduces Cross-Attentive Modulation (CAM) tokens, which are tokens whose initial value is learned, gather information through cross-attention, and modulate the nodes and edges accordingly. These tokens are meant to improve the global awareness of link predictions models which, based on graph neural networks, can struggle to capture graph-level features. This lack of ability to feature high level representations is particularly limiting when predicting multiple or entire sets of links. We implement CAM tokens in a simple attention-based link prediction model and in a graph transformer, which we also use in a denoising diffusion framework. A brief introduction to our toy datasets will then be followed by benchmarks which prove that CAM token improve the performance of the model they supplement and outperform a baseline with diverse statistical graph attributes. |
| title | Improving global awareness of linkset predictions using Cross-Attentive Modulation tokens |
| topic | Social and Information Networks Machine Learning I.2.6 |
| url | https://arxiv.org/abs/2405.19375 |