Image de couverture de livre

On the token distance modeling ability of higher RoPE attention dimension

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Hong, Xiangyu, Jiang, Che, Qi, Biqing, Meng, Fandong, Yu, Mo, Zhou, Bowen, Zhou, Jie
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

RoSA: Enhancing Parameter-Efficient Fine-Tuning via RoPE-aware Selective Adaptation in Large Language Models
par: Pan, Dayan, et autres
Publié: (2025)

Periodic RoPE for Infinite Context LLMs
par: Huo, Simin
Publié: (2026)

Scaling Laws of RoPE-based Extrapolation
par: Liu, Xiaoran, et autres
Publié: (2023)

On Large Language Models' Hallucination with Regard to Known Facts
par: Jiang, Che, et autres
Publié: (2024)

Demystifying the Slash Pattern in Attention: The Role of RoPE
par: Cheng, Yuan, et autres
Publié: (2026)

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
par: Yang, Ning, et autres
Publié: (2026)

RoPE-LIME: RoPE-Space Locality + Sparse-K Sampling for Efficient LLM Attribution
par: Picov, Isaac, et autres
Publié: (2026)

RoPE Attention Can Be Trained in Almost Linear Time
par: Cao, Yang, et autres
Publié: (2024)

Circuit Complexity Bounds for RoPE-based Transformer Architecture
par: Chen, Bo, et autres
Publié: (2024)

Resonance RoPE: Improving Context Length Generalization of Large Language Models
par: Wang, Suyuchen, et autres
Publié: (2024)

CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
par: Li, Haoran, et autres
Publié: (2026)

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
par: Du, Yufeng, et autres
Publié: (2026)

Base of RoPE Bounds Context Length
par: Men, Xin, et autres
Publié: (2024)

EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
par: Zhou, Yuhao, et autres
Publié: (2025)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
par: Li, Xiaoyu, et autres
Publié: (2024)

MSI-Agent: Incorporating Multi-Scale Insight into Embodied Agents for Superior Planning and Decision-Making
par: Fu, Dayuan, et autres
Publié: (2024)

Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
par: Liu, Feilong
Publié: (2026)

Frayed RoPE and Long Inputs: A Geometric Perspective
par: Wertheimer, Davis, et autres
Publié: (2026)

CSCD-NS: a Chinese Spelling Check Dataset for Native Speakers
par: Hu, Yong, et autres
Publié: (2022)

DeepTrans: Deep Reasoning Translation via Reinforcement Learning
par: Wang, Jiaan, et autres
Publié: (2025)

ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learning
par: Wang, Jiaan, et autres
Publié: (2025)

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
par: Zhong, Meizhi, et autres
Publié: (2024)

Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding
par: Liu, Haiping, et autres
Publié: (2025)

DePass: Unified Feature Attributing by Simple Decomposed Forward Pass
par: Hong, Xiangyu, et autres
Publié: (2025)

Beyond Next Token Prediction: Patch-Level Training for Large Language Models
par: Shao, Chenze, et autres
Publié: (2024)

Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
par: Li, Zichong, et autres
Publié: (2026)

HGMEM: Hypergraph-based Working Memory to Improve Multi-step RAG for Long-Context Complex Relational Modeling
par: Zhou, Chulun, et autres
Publié: (2025)

ReRoPE: Repurposing RoPE for Relative Camera Control
par: Li, Chunyang, et autres
Publié: (2026)

Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE
par: Khan, Mohammad Aflah, et autres
Publié: (2026)

Retrieval-Augmented Machine Translation with Unstructured Knowledge
par: Wang, Jiaan, et autres
Publié: (2024)

EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation
par: Xu, Yulin, et autres
Publié: (2022)

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
par: Yang, Zhen, et autres
Publié: (2023)

DRT: Deep Reasoning Translation via Long Chain-of-Thought
par: Wang, Jiaan, et autres
Publié: (2024)

RAP: KV-Cache Compression via RoPE-Aligned Pruning
par: Xin, Jihao, et autres
Publié: (2026)

A Circular Argument : Does RoPE need to be Equivariant for Vision?
par: van de Geijn, Chase, et autres
Publié: (2025)

Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation
par: Xu, Shicheng, et autres
Publié: (2024)

Continuous Autoregressive Language Models
par: Shao, Chenze, et autres
Publié: (2025)

Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks
par: Zhang, Yaobo
Publié: (2026)

When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
par: Wang, Haonan, et autres
Publié: (2024)

General learned delegation by clones
par: Li, Darren, et autres
Publié: (2026)