Cover Image

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yang, Ning, Zhong, Hengyu, Wang, Wentao, Tian, Baoliang, Zhang, Haijun, Wang, Jun
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

RoPE Attention Can Be Trained in Almost Linear Time
by: Cao, Yang, et al.
Published: (2024)

Demystifying the Slash Pattern in Attention: The Role of RoPE
by: Cheng, Yuan, et al.
Published: (2026)

Scaling Laws of RoPE-based Extrapolation
by: Liu, Xiaoran, et al.
Published: (2023)

Periodic RoPE for Infinite Context LLMs
by: Huo, Simin
Published: (2026)

Resonance RoPE: Improving Context Length Generalization of Large Language Models
by: Wang, Suyuchen, et al.
Published: (2024)

On the token distance modeling ability of higher RoPE attention dimension
by: Hong, Xiangyu, et al.
Published: (2024)

CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
by: Li, Haoran, et al.
Published: (2026)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
by: Li, Xiaoyu, et al.
Published: (2024)

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
by: Du, Yufeng, et al.
Published: (2026)

Circuit Complexity Bounds for RoPE-based Transformer Architecture
by: Chen, Bo, et al.
Published: (2024)

RoSA: Enhancing Parameter-Efficient Fine-Tuning via RoPE-aware Selective Adaptation in Large Language Models
by: Pan, Dayan, et al.
Published: (2025)

Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
by: Liu, Feilong
Published: (2026)

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
by: Zhong, Meizhi, et al.
Published: (2024)

EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
by: Zhou, Yuhao, et al.
Published: (2025)

Base of RoPE Bounds Context Length
by: Men, Xin, et al.
Published: (2024)

RoPE-LIME: RoPE-Space Locality + Sparse-K Sampling for Efficient LLM Attribution
by: Picov, Isaac, et al.
Published: (2026)

Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
by: Li, Zichong, et al.
Published: (2026)

A Circular Argument : Does RoPE need to be Equivariant for Vision?
by: van de Geijn, Chase, et al.
Published: (2025)

Frayed RoPE and Long Inputs: A Geometric Perspective
by: Wertheimer, Davis, et al.
Published: (2026)

ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Models
by: Li, Bozhou, et al.
Published: (2025)

Higher-order Linear Attention
by: Zhang, Yifan, et al.
Published: (2025)

RAP: KV-Cache Compression via RoPE-Aligned Pruning
by: Xin, Jihao, et al.
Published: (2026)

Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
by: Chen, Yingfa, et al.
Published: (2026)

RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers
by: Gokmen, Ahmet Berke, et al.
Published: (2025)

LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modeling
by: Liu, Zeyu, et al.
Published: (2025)

RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale
by: Goldstein, Daniel, et al.
Published: (2025)

Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
by: Zhu, Peijun, et al.
Published: (2025)

GuARD: Effective Anomaly Detection through a Text-Rich and Graph-Informed Language Model
by: Pang, Yunhe, et al.
Published: (2024)

Sycophancy Hides Linearly in the Attention Heads
by: Genadi, Rifo, et al.
Published: (2026)

Untangling Component Imbalance in Hybrid Linear Attention Conversion Methods
by: Benfeghoul, Martin, et al.
Published: (2025)

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
by: Urrutia, Felipe, et al.
Published: (2026)

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
by: MiniCPM Team, et al.
Published: (2026)

Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention
by: Weng, Yuzhe, et al.
Published: (2024)

LASP-2: Rethinking Sequence Parallelism for Linear Attention and Its Hybrid
by: Sun, Weigao, et al.
Published: (2025)

Parallax: Parameterized Local Linear Attention for Language Modeling
by: Zuo, Yifei, et al.
Published: (2026)

When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
by: You, Haoran, et al.
Published: (2024)

Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements
by: Haller, Patrick, et al.
Published: (2025)

Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding
by: Liu, Haiping, et al.
Published: (2025)

Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
by: Li, Shuaiyi, et al.
Published: (2026)

MGSA: Multi-Granularity Graph Structure Attention for Knowledge Graph-to-Text Generation
by: Wang, Shanshan, et al.
Published: (2024)