Cover Image

KV Shifting Attention Enhances Language Modeling

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xu, Mingyu, Cheng, Wei, Wang, Bingning, Chen, Weipeng
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic
by: Zhou, Yuyan, et al.
Published: (2024)

ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
by: Men, Xin, et al.
Published: (2024)

LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
by: Dong, Zican, et al.
Published: (2025)

Full-ECE: A Metric For Token-level Calibration on Large Language Models
by: Liu, Han, et al.
Published: (2024)

Base of RoPE Bounds Context Length
by: Men, Xin, et al.
Published: (2024)

Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
by: Chen, Jie, et al.
Published: (2024)

Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models
by: Chen, Zhipeng, et al.
Published: (2024)

Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFN
by: Xu, Yao, et al.
Published: (2025)

AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models
by: Su, Zunhai, et al.
Published: (2025)

AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models
by: Gu, Yifeng, et al.
Published: (2025)

X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
by: Li, Guihong, et al.
Published: (2025)

Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads
by: He, Xingyang, et al.
Published: (2025)

Checkpoint Merging via Bayesian Optimization in LLM Pretraining
by: Liu, Deyuan, et al.
Published: (2024)

Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models
by: Duan, Jinhao, et al.
Published: (2023)

WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models
by: Yuan, Jian, et al.
Published: (2025)

KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
by: Su, Zunhai, et al.
Published: (2025)

G-KV: Decoding-Time KV Cache Eviction with Global Attention
by: Liao, Mengqi, et al.
Published: (2025)

A Method for Building Large Language Models with Predefined KV Cache Capacity
by: Yi, Zhonghua, et al.
Published: (2024)

AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
by: Yang, Qingyue, et al.
Published: (2025)

dKV-Cache: The Cache for Diffusion Language Models
by: Ma, Xinyin, et al.
Published: (2025)

Beyond Filtering: Adaptive Image-Text Quality Enhancement for MLLM Pretraining
by: Huang, Han, et al.
Published: (2024)

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression
by: Mao, Weian, et al.
Published: (2026)

QAQ: Quality Adaptive Quantization for LLM KV Cache
by: Dong, Shichen, et al.
Published: (2024)

KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
by: Yang, Jingbo, et al.
Published: (2025)

KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
by: Rehg, Isaac
Published: (2024)

$A^3$: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving
by: Zhou, Yuechi, et al.
Published: (2025)

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
by: Ma, Da, et al.
Published: (2024)

Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers
by: Lin, Hongzhan, et al.
Published: (2025)

Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
by: Song, Guanghui, et al.
Published: (2025)

Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
by: Jin, Mingyu, et al.
Published: (2026)

RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Eviction
by: Jiang, Tanqiu, et al.
Published: (2024)

YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference
by: Wu, You, et al.
Published: (2026)

HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
by: Gao, Yizhao, et al.
Published: (2026)

KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
by: Chen, Jian, et al.
Published: (2026)

CLIMB: A Benchmark of Clinical Bias in Large Language Models
by: Zhang, Yubo, et al.
Published: (2024)

RefreshKV: Updating Small KV Cache During Long-form Generation
by: Xu, Fangyuan, et al.
Published: (2024)

TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
by: Yao, Dingyu, et al.
Published: (2025)

WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
by: Zuo, Youhui, et al.
Published: (2025)

CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
by: Song, Jiwon, et al.
Published: (2026)

R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
by: Cai, Zefan, et al.
Published: (2025)