KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918144415629312 |
|---|---|
| author | Akulov, Dmitry Sana, Mohamed De Domenico, Antonio Salem, Tareq Si Piovesan, Nicola Ayed, Fadhel |
| author_facet | Akulov, Dmitry Sana, Mohamed De Domenico, Antonio Salem, Tareq Si Piovesan, Nicola Ayed, Fadhel |
| contents | Large language models (LLMs) rely on key-value (KV) caches for efficient autoregressive decoding; however, cache size grows linearly with context length and model depth, becoming a major bottleneck in long-context inference. Prior KV cache compression methods either enforce rigid heuristics, disrupt tensor layouts with per-attention-head variability, or require specialized compute kernels.
We propose a simple, yet effective, KV cache compression framework based on attention-guided, layer-adaptive composite tokens. Our method aggregates attention scores to estimate token importance, selects head-specific tokens independently, and aligns them into composite tokens that respect the uniform cache structure required by existing inference engines. A global allocation mechanism further adapts retention budgets across layers, assigning more capacity to layers with informative tokens. This approach achieves significant memory reduction while preserving accuracy, consistently outperforming prior structured and semi-structured methods. Crucially, our approach remains fully compatible with standard inference pipelines, offering a practical and scalable solution for efficient long-context LLM deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_05165 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KVCompose: Efficient Structured KV Cache Compression with Composite Tokens Akulov, Dmitry Sana, Mohamed De Domenico, Antonio Salem, Tareq Si Piovesan, Nicola Ayed, Fadhel Machine Learning Large language models (LLMs) rely on key-value (KV) caches for efficient autoregressive decoding; however, cache size grows linearly with context length and model depth, becoming a major bottleneck in long-context inference. Prior KV cache compression methods either enforce rigid heuristics, disrupt tensor layouts with per-attention-head variability, or require specialized compute kernels. We propose a simple, yet effective, KV cache compression framework based on attention-guided, layer-adaptive composite tokens. Our method aggregates attention scores to estimate token importance, selects head-specific tokens independently, and aligns them into composite tokens that respect the uniform cache structure required by existing inference engines. A global allocation mechanism further adapts retention budgets across layers, assigning more capacity to layers with informative tokens. This approach achieves significant memory reduction while preserving accuracy, consistently outperforming prior structured and semi-structured methods. Crucially, our approach remains fully compatible with standard inference pipelines, offering a practical and scalable solution for efficient long-context LLM deployment. |
| title | KVCompose: Efficient Structured KV Cache Compression with Composite Tokens |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.05165 |