PaTH Attention: Position Encoding via Accumulating Householder Transformations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912874103832576 |
|---|---|
| author | Yang, Songlin Shen, Yikang Wen, Kaiyue Tan, Shawn Mishra, Mayank Ren, Liliang Panda, Rameswar Kim, Yoon |
| author_facet | Yang, Songlin Shen, Yikang Wen, Kaiyue Tan, Shawn Mishra, Mayank Ren, Liliang Panda, Rameswar Kim, Yoon |
| contents | The attention mechanism is a core primitive in modern large language models (LLMs) and AI more broadly. Since attention by itself is permutation-invariant, position encoding is essential for modeling structured domains such as language. Rotary position encoding (RoPE) has emerged as the de facto standard approach for position encoding and is part of many modern LLMs. However, in RoPE the key/query transformation between two elements in a sequence is only a function of their relative position and otherwise independent of the actual input. This limits the expressivity of RoPE-based transformers.
This paper describes PaTH, a flexible data-dependent position encoding scheme based on accumulated products of Householder(like) transformations, where each transformation is data-dependent, i.e., a function of the input. We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices, and implement a FlashAttention-style blockwise algorithm. Across both targeted synthetic benchmarks and moderate-scale real-world language modeling experiments, we find that PaTH improves upon RoPE and other recent baselines. Finally, we show that we can convert pretrained RoPE transformers into PaTH with continued pretraining. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16381 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PaTH Attention: Position Encoding via Accumulating Householder Transformations Yang, Songlin Shen, Yikang Wen, Kaiyue Tan, Shawn Mishra, Mayank Ren, Liliang Panda, Rameswar Kim, Yoon Computation and Language Machine Learning The attention mechanism is a core primitive in modern large language models (LLMs) and AI more broadly. Since attention by itself is permutation-invariant, position encoding is essential for modeling structured domains such as language. Rotary position encoding (RoPE) has emerged as the de facto standard approach for position encoding and is part of many modern LLMs. However, in RoPE the key/query transformation between two elements in a sequence is only a function of their relative position and otherwise independent of the actual input. This limits the expressivity of RoPE-based transformers. This paper describes PaTH, a flexible data-dependent position encoding scheme based on accumulated products of Householder(like) transformations, where each transformation is data-dependent, i.e., a function of the input. We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices, and implement a FlashAttention-style blockwise algorithm. Across both targeted synthetic benchmarks and moderate-scale real-world language modeling experiments, we find that PaTH improves upon RoPE and other recent baselines. Finally, we show that we can convert pretrained RoPE transformers into PaTH with continued pretraining. |
| title | PaTH Attention: Position Encoding via Accumulating Householder Transformations |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.16381 |