Cover Image

Demystifying the Slash Pattern in Attention: The Role of RoPE

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Cheng, Yuan, Zhang, Fengzhuo, Hou, Yunlong, Du, Cunxiao, Du, Chao, Pang, Tianyu, Sun, Aixin, Yang, Zhuoran
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification
by: Yang, Penghui, et al.
Published: (2025)

When Attention Sink Emerges in Language Models: An Empirical View
by: Gu, Xiangming, et al.
Published: (2024)

When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
by: Wang, Haonan, et al.
Published: (2024)

LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation
by: Zhang, Xuan, et al.
Published: (2024)

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
by: Yang, Ning, et al.
Published: (2026)

RoPE Attention Can Be Trained in Almost Linear Time
by: Cao, Yang, et al.
Published: (2024)

Periodic RoPE for Infinite Context LLMs
by: Huo, Simin
Published: (2026)

Scaling Laws of RoPE-based Extrapolation
by: Liu, Xiaoran, et al.
Published: (2023)

BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms
by: Hou, Yunlong, et al.
Published: (2025)

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
by: Du, Yufeng, et al.
Published: (2026)

On the token distance modeling ability of higher RoPE attention dimension
by: Hong, Xiangyu, et al.
Published: (2024)

Resonance RoPE: Improving Context Length Generalization of Large Language Models
by: Wang, Suyuchen, et al.
Published: (2024)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
by: Li, Xiaoyu, et al.
Published: (2024)

RoSA: Enhancing Parameter-Efficient Fine-Tuning via RoPE-aware Selective Adaptation in Large Language Models
by: Pan, Dayan, et al.
Published: (2025)

Circuit Complexity Bounds for RoPE-based Transformer Architecture
by: Chen, Bo, et al.
Published: (2024)

CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
by: Li, Haoran, et al.
Published: (2026)

Muon Outperforms Adam in Tail-End Associative Memory Learning
by: Wang, Shuche, et al.
Published: (2025)

Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
by: Liu, Feilong
Published: (2026)

Annealed Relaxation of Speculative Decoding for Faster Autoregressive Image Generation
by: Li, Xingyao, et al.
Published: (2026)

EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
by: Zhou, Yuhao, et al.
Published: (2025)

From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems
by: He, Jianliang, et al.
Published: (2024)

RoPE-LIME: RoPE-Space Locality + Sparse-K Sampling for Efficient LLM Attribution
by: Picov, Isaac, et al.
Published: (2026)

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
by: Urrutia, Felipe, et al.
Published: (2026)

Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding
by: Liu, Haiping, et al.
Published: (2025)

Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods
by: Hu, Xinyang, et al.
Published: (2024)

RAP: KV-Cache Compression via RoPE-Aligned Pruning
by: Xin, Jihao, et al.
Published: (2026)

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
by: Zhong, Meizhi, et al.
Published: (2024)

A Circular Argument : Does RoPE need to be Equivariant for Vision?
by: van de Geijn, Chase, et al.
Published: (2025)

Base of RoPE Bounds Context Length
by: Men, Xin, et al.
Published: (2024)

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
by: Tian, Yuandong, et al.
Published: (2023)

Scalable Token-Level Hallucination Detection in Large Language Models
by: Min, Rui, et al.
Published: (2026)

RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers
by: Gokmen, Ahmet Berke, et al.
Published: (2025)

A Closer Look at Machine Unlearning for Large Language Models
by: Yuan, Xiaojian, et al.
Published: (2024)

Adaptive 3D-RoPE: Physics-Aligned Rotary Positional Encoding for Wireless Foundation Models
by: Zhang, Chenyu, et al.
Published: (2026)

Rethinking RoPE Scaling in Quantized LLM: Theory, Outlier, and Channel-Band Analysis with Weight Rescaling
by: Qiao, Ye, et al.
Published: (2025)

Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs
by: Qiao, Ye, et al.
Published: (2025)

LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
by: Huang, Chengsong, et al.
Published: (2023)

Optimizing Anytime Reasoning via Budget Relative Policy Optimization
by: Qi, Penghui, et al.
Published: (2025)

Frayed RoPE and Long Inputs: A Geometric Perspective
by: Wertheimer, Davis, et al.
Published: (2026)

Improving Your Model Ranking on Chatbot Arena by Vote Rigging
by: Min, Rui, et al.
Published: (2025)