Rotary Position Encodings for Graphs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914290588450816 |
|---|---|
| author | Reid, Isaac Sehanobish, Arijit Höfs, Cederik Mlodozeniec, Bruno Vulpius, Leonhard Barbero, Federico Weller, Adrian Choromanski, Krzysztof Turner, Richard E. Veličković, Petar |
| author_facet | Reid, Isaac Sehanobish, Arijit Höfs, Cederik Mlodozeniec, Bruno Vulpius, Leonhard Barbero, Federico Weller, Adrian Choromanski, Krzysztof Turner, Richard E. Veličković, Petar |
| contents | We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficiently injects structural information into the attention mechanism, boosting performance in synthetic and real-world graph learning tasks. This approach, coined _Wave-Induced Rotary Encodings_ (WIRE), enjoys intriguing theoretical properties: it recovers regular RoPE on grids, and depends asymptotically on the graph effective resistance. Unlike bias-based relative position encodings, WIRE is compatible with linear attention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_22259 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rotary Position Encodings for Graphs Reid, Isaac Sehanobish, Arijit Höfs, Cederik Mlodozeniec, Bruno Vulpius, Leonhard Barbero, Federico Weller, Adrian Choromanski, Krzysztof Turner, Richard E. Veličković, Petar Machine Learning Artificial Intelligence We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transformers (ViTs), can be applied to graph-structured data. We find that rotating tokens depending on the spectrum of the graph Laplacian efficiently injects structural information into the attention mechanism, boosting performance in synthetic and real-world graph learning tasks. This approach, coined _Wave-Induced Rotary Encodings_ (WIRE), enjoys intriguing theoretical properties: it recovers regular RoPE on grids, and depends asymptotically on the graph effective resistance. Unlike bias-based relative position encodings, WIRE is compatible with linear attention. |
| title | Rotary Position Encodings for Graphs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.22259 |