Cover Image

Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHA

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Graef, Nils, Wasielewski, Andrew
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

KV-weights are all you need for skipless transformers
by: Graef, Nils
Published: (2024)

FlashNorm: Fast Normalization for Transformers
by: Graef, Nils, et al.
Published: (2024)

Linear attention is (maybe) all you need (to understand transformer optimization)
by: Ahn, Kwangjun, et al.
Published: (2023)

Transformer tricks: Precomputing the first layer
by: Graef, Nils
Published: (2024)

Is attention all you need in medical image analysis? A review
by: Papanastasiou, Giorgos, et al.
Published: (2023)

One protein is all you need
by: Bushuiev, Anton, et al.
Published: (2024)

Graph is all you need? Lightweight data-agnostic neural architecture search without training
by: Huang, Zhenhan, et al.
Published: (2024)

Kolmogorov GAM Networks are all you need!
by: Polson, Sarah, et al.
Published: (2025)

Tabular Data: Is Deep Learning all you need?
by: Zabërgja, Guri, et al.
Published: (2024)

Attention and Compression is all you need for Controllably Efficient Language Models
by: Prakash, Jatin, et al.
Published: (2025)

Image compositing is all you need for data augmentation
by: Shermaine, Ang Jia Ning, et al.
Published: (2025)

Large Language Models aren't all that you need
by: Holla, Kiran Voderhobli, et al.
Published: (2024)

Experts are all you need: A Composable Framework for Large Language Model Inference
by: Sridharan, Shrihari, et al.
Published: (2025)

Addition is almost all you need: Compressing large language models with double binary factorization
by: Boža, Vladimír, et al.
Published: (2025)

Cross-Modal Safety Alignment: Is textual unlearning all you need?
by: Chakraborty, Trishna, et al.
Published: (2024)

Attention is all you need for boosting graph convolutional neural network
by: Wu, Yinwei
Published: (2024)

BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
by: Gulhan, Ahmed Burak, et al.
Published: (2025)

Self-attention as an attractor network: transient memories without backpropagation
by: D'Amico, Francesco, et al.
Published: (2024)

Simulation-based inference with scattering representations: scattering is all you need
by: Lin, Kiyam, et al.
Published: (2024)

DC is all you need: describing ReLU from a signal processing standpoint
by: Kechris, Christodoulos, et al.
Published: (2024)

Few Labels are all you need: A Weakly Supervised Framework for Appliance Localization in Smart-Meter Series
by: Petralia, Adrien, et al.
Published: (2025)

In-context denoising with one-layer transformers: connections between attention and associative memory retrieval
by: Smart, Matthew, et al.
Published: (2025)

Why you don't overfit, and don't need Bayes if you only train for one epoch
by: Aitchison, Laurence
Published: (2024)

Signformer is all you need: Towards Edge AI for Sign Language
by: Yang, Eta
Published: (2024)

An extension of linear self-attention for in-context learning
by: Hagiwara, Katsuyuki
Published: (2025)

On Surjectivity of Neural Networks: Can you elicit any behavior from your model?
by: Jiang, Haozhe, et al.
Published: (2025)

An experimental study of KV cache reuse strategies in chunk-level caching systems
by: Cestola, Samuel, et al.
Published: (2026)

Neural Operator: Is data all you need to model the world? An insight into the paradigm of data-driven scientific ML
by: Viswanath, Hrishikesh, et al.
Published: (2023)

Attention is all you need for an improved CNN-based flash flood susceptibility modeling. The case of the ungauged Rheraya watershed, Morocco
by: Elghouat, Akram, et al.
Published: (2024)

KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems
by: Ye, Hancheng, et al.
Published: (2025)

SALS: Sparse Attention in Latent Space for KV cache Compression
by: Mu, Junlin, et al.
Published: (2025)

What are you sinking? A geometric approach on attention sink
by: Ruscio, Valeria, et al.
Published: (2025)

SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
by: Li, Zichong, et al.
Published: (2025)

100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances
by: Pacchiardi, Lorenzo, et al.
Published: (2024)

An Innovative CGL-MHA Model for Sarcasm Sentiment Recognition Using the MindSpore Framework
by: Qin, Zhenkai, et al.
Published: (2024)

SlimDiff: Training-Free, Activation-Guided Hands-free Slimming of Diffusion Models
by: Roy, Arani, et al.
Published: (2025)

Residual vector quantization for KV cache compression in large language model
by: Kumar, Ankur
Published: (2024)

TransformerFAM: Feedback attention is working memory
by: Hwang, Dongseong, et al.
Published: (2024)

Can you Finetune your Binoculars? Embedding Text Watermarks into the Weights of Large Language Models
by: Elhassan, Fay, et al.
Published: (2025)

Don't lie to your friends: Learning what you know from collaborative self-play
by: Eisenstein, Jacob, et al.
Published: (2025)