Cover Image

Round and Round We Go! What makes Rotary Positional Encodings useful?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Barbero, Federico, Vitvitskyi, Alex, Perivolaropoulos, Christos, Pascanu, Razvan, Veličković, Petar
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Perplexity Cannot Always Tell Right from Wrong
by: Veličković, Petar, et al.
Published: (2026)

Softmax is not Enough (for Sharp Size Generalisation)
by: Veličković, Petar, et al.
Published: (2024)

Transformers need glasses! Information over-squashing in language tasks
by: Barbero, Federico, et al.
Published: (2024)

Filter Equivariant Functions: A symmetric account of length-general extrapolation on lists
by: Lewis, Owen, et al.
Published: (2025)

Transformers meet Neural Algorithmic Reasoners
by: Bounsi, Wilfried, et al.
Published: (2024)

Why do LLMs attend to the first token?
by: Barbero, Federico, et al.
Published: (2025)

Mining Generalizable Activation Functions
by: Vitvitskyi, Alex, et al.
Published: (2026)

The Illusion of Stochasticity in LLMs
by: Gu, Xiangming, et al.
Published: (2026)

Rotary Position Encodings for Graphs
by: Reid, Isaac, et al.
Published: (2025)

What makes a good feedforward computational graph?
by: Vitvitskyi, Alex, et al.
Published: (2025)

Latent Space Representations of Neural Algorithmic Reasoners
by: Mirjanić, Vladimir V., et al.
Published: (2023)

Amplifying human performance in combinatorial competitive programming
by: Veličković, Petar, et al.
Published: (2024)

How do LLMs Compute Verbal Confidence
by: Kumaran, Dharshan, et al.
Published: (2026)

Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Models
by: Gu, Xiangming, et al.
Published: (2026)

Selective Rotary Position Embedding
by: Movahedi, Sajad, et al.
Published: (2025)

Asynchronous Algorithmic Alignment with Cocycles
by: Dudzik, Andrew, et al.
Published: (2023)

Direct Quantized Training of Language Models with Stochastic Rounding
by: Zhao, Kaiyan, et al.
Published: (2024)

Revisiting Adaptive Rounding with Vectorized Reparameterization for LLM Quantization
by: Zhou, Yuli, et al.
Published: (2026)

The CLRS-Text Algorithmic Reasoning Language Benchmark
by: Markeeva, Larisa, et al.
Published: (2024)

How do language models learn facts? Dynamics, curricula and hallucinations
by: Zucchet, Nicolas, et al.
Published: (2025)

Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Tasks
by: Dong, Wenhan, et al.
Published: (2025)

Optimizers Qualitatively Alter Solutions And We Should Leverage This
by: Pascanu, Razvan, et al.
Published: (2025)

Trust in One Round: Confidence Estimation for Large Language Models via Structural Signals
by: Yang, Pengyue, et al.
Published: (2026)

Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks
by: Cheng, Yixin, et al.
Published: (2024)

Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
by: Cheng, Wenhua, et al.
Published: (2023)

Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation
by: Ruiz, Fernando Vallecillos, et al.
Published: (2024)

The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval
by: Chiang, Ting-Rui, et al.
Published: (2025)

On the Geometry of Positional Encodings in Transformers
by: Cirrincione, Giansalvo
Published: (2026)

Towards Understanding Multi-Round Large Language Model Reasoning: Approximability, Learnability and Generalizability
by: Xu, Chenhui, et al.
Published: (2025)

Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning
by: Zhang, Haozhen, et al.
Published: (2025)

ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
by: Chen, Justin Chih-Yao, et al.
Published: (2023)

Rotary Offset Features in Large Language Models
by: Jonasson, André
Published: (2025)

Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models
by: Rannen-Triki, Amal, et al.
Published: (2024)

Group Representational Position Encoding
by: Zhang, Yifan, et al.
Published: (2025)

Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
by: Liu, Feilong
Published: (2026)

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings
by: Zuo, Chunsheng, et al.
Published: (2024)

Fusion Matters: Length-Aware Analysis of Positional-Encoding Fusion in Transformers
by: Hallam, Mohamed Amine, et al.
Published: (2026)

PaTH Attention: Position Encoding via Accumulating Householder Transformations
by: Yang, Songlin, et al.
Published: (2025)

Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph
by: Irie, Kazuki
Published: (2024)

On the generalization of language models from in-context learning and finetuning: a controlled study
by: Lampinen, Andrew K., et al.
Published: (2025)