Cover Image

Base of RoPE Bounds Context Length

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Men, Xin, Xu, Mingyu, Wang, Bingning, Zhang, Qingyu, Lin, Hongyu, Han, Xianpei, Chen, Weipeng
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
by: Men, Xin, et al.
Published: (2024)

Resonance RoPE: Improving Context Length Generalization of Large Language Models
by: Wang, Suyuchen, et al.
Published: (2024)

Periodic RoPE for Infinite Context LLMs
by: Huo, Simin
Published: (2026)

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
by: Zhong, Meizhi, et al.
Published: (2024)

RoPE-LIME: RoPE-Space Locality + Sparse-K Sampling for Efficient LLM Attribution
by: Picov, Isaac, et al.
Published: (2026)

Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
by: Li, Zichong, et al.
Published: (2026)

Scaling Laws of RoPE-based Extrapolation
by: Liu, Xiaoran, et al.
Published: (2023)

Circuit Complexity Bounds for RoPE-based Transformer Architecture
by: Chen, Bo, et al.
Published: (2024)

Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
by: Liu, Feilong
Published: (2026)

CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
by: Li, Haoran, et al.
Published: (2026)

Frayed RoPE and Long Inputs: A Geometric Perspective
by: Wertheimer, Davis, et al.
Published: (2026)

KV Shifting Attention Enhances Language Modeling
by: Xu, Mingyu, et al.
Published: (2024)

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
by: Yang, Ning, et al.
Published: (2026)

Demystifying the Slash Pattern in Attention: The Role of RoPE
by: Cheng, Yuan, et al.
Published: (2026)

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
by: Du, Yufeng, et al.
Published: (2026)

When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
by: Wang, Haonan, et al.
Published: (2024)

RoPE Attention Can Be Trained in Almost Linear Time
by: Cao, Yang, et al.
Published: (2024)

ReRoPE: Repurposing RoPE for Relative Camera Control
by: Li, Chunyang, et al.
Published: (2026)

On the token distance modeling ability of higher RoPE attention dimension
by: Hong, Xiangyu, et al.
Published: (2024)

Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE
by: Khan, Mohammad Aflah, et al.
Published: (2026)

Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks
by: Zhang, Yaobo
Published: (2026)

LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
by: Dong, Zican, et al.
Published: (2025)

RoSA: Enhancing Parameter-Efficient Fine-Tuning via RoPE-aware Selective Adaptation in Large Language Models
by: Pan, Dayan, et al.
Published: (2025)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
by: Li, Xiaoyu, et al.
Published: (2024)

ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Models
by: Li, Bozhou, et al.
Published: (2025)

LongRoPE2: Near-Lossless LLM Context Window Scaling
by: Shang, Ning, et al.
Published: (2025)

MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic
by: Zhou, Yuyan, et al.
Published: (2024)

Full-ECE: A Metric For Token-level Calibration on Large Language Models
by: Liu, Han, et al.
Published: (2024)

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
by: Ding, Yiran, et al.
Published: (2024)

EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
by: Zhou, Yuhao, et al.
Published: (2025)

Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
by: Chen, Jie, et al.
Published: (2024)

AI-Salesman: Towards Reliable Large Language Model Driven Telemarketing
by: Zhang, Qingyu, et al.
Published: (2025)

Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers
by: Xin, Rihui, et al.
Published: (2025)

The Life Cycle of Knowledge in Big Language Models: A Survey
by: Cao, Boxi, et al.
Published: (2023)

Academically intelligent LLMs are not necessarily socially intelligent
by: Xu, Ruoxi, et al.
Published: (2024)

Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation
by: Pan, Ruotong, et al.
Published: (2024)

Exploring the Design Space of Visual Context Representation in Video MLLMs
by: Du, Yifan, et al.
Published: (2024)

ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers
by: Yuan, Qianhao, et al.
Published: (2025)

MrRoPE: Mixed-radix Rotary Position Embedding
by: Tian, Qingyuan, et al.
Published: (2026)

Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models
by: Chen, Zhipeng, et al.
Published: (2024)