Image de couverture de livre

Q-ROAR: Outlier-Aware Rescaling for RoPE Position Interpolation in Quantized Long-Context LLMs

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Qiao, Ye, Huang, Sitao
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Rethinking RoPE Scaling in Quantized LLM: Theory, Outlier, and Channel-Band Analysis with Weight Rescaling
par: Qiao, Ye, et autres
Publié: (2025)

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
par: Du, Yufeng, et autres
Publié: (2026)

CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
par: Li, Haoran, et autres
Publié: (2026)

Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
par: Liu, Feilong
Publié: (2026)

Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embedding
par: Liu, Haiping, et autres
Publié: (2025)

Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
par: Urrutia, Felipe, et autres
Publié: (2026)

Demystifying the Slash Pattern in Attention: The Role of RoPE
par: Cheng, Yuan, et autres
Publié: (2026)

RAP: KV-Cache Compression via RoPE-Aligned Pruning
par: Xin, Jihao, et autres
Publié: (2026)

Periodic RoPE for Infinite Context LLMs
par: Huo, Simin
Publié: (2026)

RoPE Attention Can Be Trained in Almost Linear Time
par: Cao, Yang, et autres
Publié: (2024)

Circuit Complexity Bounds for RoPE-based Transformer Architecture
par: Chen, Bo, et autres
Publié: (2024)

RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers
par: Gokmen, Ahmet Berke, et autres
Publié: (2025)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
par: Li, Xiaoyu, et autres
Publié: (2024)

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization
par: Huang, Xijie, et autres
Publié: (2024)

EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
par: Zhou, Yuhao, et autres
Publié: (2025)

FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression
par: Qiao, Ye, et autres
Publié: (2026)

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
par: Schenck, Connor, et autres
Publié: (2025)

MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantization
par: Ramachandran, Akshat, et autres
Publié: (2024)

MicroNAS: Zero-Shot Neural Architecture Search for MCUs
par: Qiao, Ye, et autres
Publié: (2024)

TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Search
par: Qiao, Ye, et autres
Publié: (2024)

Frayed RoPE and Long Inputs: A Geometric Perspective
par: Wertheimer, Davis, et autres
Publié: (2026)

Scaling Laws of RoPE-based Extrapolation
par: Liu, Xiaoran, et autres
Publié: (2023)

FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration
par: Baek, Daehyeon, et autres
Publié: (2025)

HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation
par: Chen, Yuhan, et autres
Publié: (2024)

Resonance RoPE: Improving Context Length Generalization of Large Language Models
par: Wang, Suyuchen, et autres
Publié: (2024)

HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning
par: Zhang, Jinhao Zhang Yunquan, et autres
Publié: (2026)

On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
par: Ye, Rongguang, et autres
Publié: (2025)

RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models
par: Wei, Quan, et autres
Publié: (2025)

Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs
par: Paglieri, Davide, et autres
Publié: (2024)

Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis
par: Huang, Hong, et autres
Publié: (2025)

RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations
par: Su, Zunhai, et autres
Publié: (2025)

Long-Term Outlier Prediction Through Outlier Score Modeling
par: Aoki, Yuma, et autres
Publié: (2026)

MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning
par: Zhang, Tao, et autres
Publié: (2025)

Astro: Activation-guided Structured Regularization for Outlier-Robust LLM Post-Training Quantization
par: Chen, Xi, et autres
Publié: (2026)

GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs
par: An, Selim, et autres
Publié: (2026)

Adaptive 3D-RoPE: Physics-Aligned Rotary Positional Encoding for Wireless Foundation Models
par: Zhang, Chenyu, et autres
Publié: (2026)

Widening the Gap: Exploiting LLM Quantization via Outlier Injection
par: Zhan, Xiaohua, et autres
Publié: (2026)

On the token distance modeling ability of higher RoPE attention dimension
par: Hong, Xiangyu, et autres
Publié: (2024)

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
par: Yang, Ning, et autres
Publié: (2026)

Characterizing State Space Model and Hybrid Language Model Performance with Long Context
par: Mitra, Saptarshi, et autres
Publié: (2025)