Cover Image

On the Geometry of Positional Encodings in Transformers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Cirrincione, Giansalvo
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers
by: Cirrincione, Giansalvo
Published: (2026)

INCRT: An Incremental Transformer That Determines Its Own Architecture
by: Cirrincione, Giansalvo
Published: (2026)

Hierarchical Kernel Transformer: Multi-Scale Attention with an Information-Theoretic Approximation Analysis
by: Cirrincione, Giansalvo
Published: (2026)

DDCL-INCRT: A Self-Organising Transformer with Hierarchical Prototype Structure (Theoretical Foundations)
by: Cirrincione, Giansalvo
Published: (2026)

DDCL: Deep Dual Competitive Learning: A Differentiable End-to-End Framework for Unsupervised Prototype-Based Representation Learning
by: Cirrincione, Giansalvo
Published: (2026)

Collapse-Free Prototype Readout Layer for Transformer Encoders
by: Cirrincione, Giansalvo, et al.
Published: (2026)

Learned Lyapunov Shielding for Adaptive Control
by: Cirrincione, Giansalvo, et al.
Published: (2026)

Temporal Attention for Adaptive Control of Euler-Lagrange Systems with Unobservable Memory
by: Cirrincione, Giansalvo, et al.
Published: (2026)

The geometry of BERT
by: Bonino, Matteo, et al.
Published: (2025)

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings
by: Zuo, Chunsheng, et al.
Published: (2024)

Fusion Matters: Length-Aware Analysis of Positional-Encoding Fusion in Transformers
by: Hallam, Mohamed Amine, et al.
Published: (2026)

PaTH Attention: Position Encoding via Accumulating Householder Transformations
by: Yang, Songlin, et al.
Published: (2025)

Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph
by: Irie, Kazuki
Published: (2024)

Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding
by: Zeris, Athanasios
Published: (2026)

SeqPE: Transformer with Sequential Position Encoding
by: Li, Huayang, et al.
Published: (2025)

Group Representational Position Encoding
by: Zhang, Yifan, et al.
Published: (2025)

Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
by: Zeris, Athanasios
Published: (2026)

Round and Round We Go! What makes Rotary Positional Encodings useful?
by: Barbero, Federico, et al.
Published: (2024)

Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks
by: Zhang, Yaobo
Published: (2026)

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
by: Zhang, Zhenyu, et al.
Published: (2024)

Encoding Agent Trajectories as Representations with Sequence Transformers
by: Tsiligkaridis, Athanasios, et al.
Published: (2024)

FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers
by: Wang, Sihan, et al.
Published: (2026)

Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
by: Yan, Hao, et al.
Published: (2026)

Short Data, Long Context: Distilling Positional Knowledge in Transformers
by: Huber, Patrick, et al.
Published: (2026)

Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance
by: Marbut, Anna C., et al.
Published: (2024)

SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression
by: Zhang, Jing, et al.
Published: (2024)

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
by: He, Zhenyu, et al.
Published: (2024)

PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models
by: Aggarwal, Arpit
Published: (2024)

Sensitivity-Positional Co-Localization in GQA Transformers
by: Rao, Manoj Chandrashekar
Published: (2026)

HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation
by: Chen, Yuhan, et al.
Published: (2024)

Learning-Time Encoding Shapes Unlearning in LLMs
by: Wu, Ruihan, et al.
Published: (2025)

Integrating Locality-Aware Attention with Transformers for General Geometry PDEs
by: Koh, Minsu, et al.
Published: (2025)

Position-Agnostic Pre-Projection for Transformer Attention: Nonlinear Feature Construction and Content Skip Before Q/K/V
by: Shinde, Chirag
Published: (2026)

Multilingual Language Models Encode Script Over Linguistic Structure
by: Verma, Aastha A K, et al.
Published: (2026)

Task Structure Reverses Layerwise State Encoding in Sequence Models
by: Jiang, Yuhang
Published: (2026)

Circuit Fingerprints: How Answer Tokens Encode Their Geometrical Path
by: Saurez, Andres, et al.
Published: (2026)

Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias
by: Chowdhury, Borun D
Published: (2026)

mini-vec2vec: Scaling Universal Geometry Alignment with Linear Transformations
by: Dar, Guy
Published: (2025)

Training Large Reasoning Models Efficiently via Progressive Thought Encoding
by: Zhang, Zeliang, et al.
Published: (2026)

Large Language Models Encode Semantics and Alignment in Linearly Separable Representations
by: Saglam, Baturay, et al.
Published: (2025)