Image de couverture de livre

Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Wei, Xiuying, Gulcehre, Caglar
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference
par: Wei, Xiuying, et autres
Publié: (2026)

BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers
par: Deschenaux, Justin, et autres
Publié: (2026)

Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
par: Deschenaux, Justin, et autres
Publié: (2024)

In Search for Architectures and Loss Functions in Multi-Objective Reinforcement Learning
par: Terekhov, Mikhail, et autres
Publié: (2024)

Partition Generative Modeling: Masked Modeling Without Masks
par: Deschenaux, Justin, et autres
Publié: (2025)

Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
par: Matrenok, Simon, et autres
Publié: (2025)

The Role of Deep Learning Regularizations on Actors in Offline RL
par: Tarasov, Denis, et autres
Publié: (2024)

The Diffusion Duality, Chapter II: $Ψ$-Samplers
par: Deschenaux, Justin, et autres
Publié: (2026)

SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning
par: Liao, Huanxuan, et autres
Publié: (2025)

RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
par: Wei, Xiuying, et autres
Publié: (2025)

From Markov to Laplace: How Mamba In-Context Learns Markov Chains
par: Bondaschi, Marco, et autres
Publié: (2025)

Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
par: Jo, Mingyu, et autres
Publié: (2025)

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO
par: Moalla, Skander, et autres
Publié: (2024)

HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
par: Joseph, Federico Arangath, et autres
Publié: (2024)

Simple Hierarchical Planning with Diffusion
par: Chen, Chang, et autres
Publié: (2024)

Control Tax: The Price of Keeping AI in Check
par: Terekhov, Mikhail, et autres
Publié: (2025)

Adaptive Memory Decay for Log-Linear Attention
par: Amin, Yaxita, et autres
Publié: (2026)

ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching
par: Zhao, Youpeng, et autres
Publié: (2024)

Sub-Token Routing in LoRA for Adaptation and Query-Aware KV Compression
par: Jiang, Wei, et autres
Publié: (2026)

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
par: Tang, Jiaming, et autres
Publié: (2024)

Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers
par: Wei, Xiuying, et autres
Publié: (2024)

Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis
par: Wei, Xiuying, et autres
Publié: (2024)

EntmaxKV: Support-Aware Decoding for Entmax Attention
par: Duarte, Gonçalo, et autres
Publié: (2026)

Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues
par: Orvieto, Antonio, et autres
Publié: (2023)

Gradformer: Graph Transformer with Exponential Decay
par: Liu, Chuang, et autres
Publié: (2024)

Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference
par: Joo, Donghyeon, et autres
Publié: (2025)

PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer
par: Chen, Chang, et autres
Publié: (2024)

RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction
par: Liu, Sihao, et autres
Publié: (2026)

RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning
par: Hu, Junhao, et autres
Publié: (2025)

Bayesian Optimistic Optimisation with Exponentially Decaying Regret
par: Tran-The, Hung, et autres
Publié: (2021)

Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
par: Li, Kunjun, et autres
Publié: (2025)

Regret-Optimized Portfolio Enhancement through Deep Reinforcement Learning and Future Looking Rewards
par: Karzanov, Daniil, et autres
Publié: (2025)

Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression
par: Zhang, Te, et autres
Publié: (2025)

ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition
par: Ye, Lu, et autres
Publié: (2024)

AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization
par: Tan, Yifan, et autres
Publié: (2024)

Exponential Family Attention
par: Wibisono, Kevin Christian, et autres
Publié: (2025)

Improving MLLM Training Efficiency via Stage-Aware Sparsity
par: Shi, Kean, et autres
Publié: (2025)

MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning
par: Zhang, Tao, et autres
Publié: (2025)

Self-Recognition in Language Models
par: Davidson, Tim R., et autres
Publié: (2024)

Fast KV Compaction via Attention Matching
par: Zweiger, Adam, et autres
Publié: (2026)