Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908903448510464 |
|---|---|
| author | Miao, Ruijie Wang, Zhiming Li, Wang Wu, Shiwei Liu, Shufan Jiang, Yanbing Yang, Tong |
| author_facet | Miao, Ruijie Wang, Zhiming Li, Wang Wu, Shiwei Liu, Shufan Jiang, Yanbing Yang, Tong |
| contents | Key-value (KV) caching is widely used to accelerate transformer inference, but its memory cost grows linearly with input length, limiting long-context deployment. Existing token eviction methods reduce memory by discarding less important tokens, which can be viewed as a coarse form of dimensionality reduction that assigns each token either zero or full dimension. We propose MixedDimKV, a mixed-dimension KV cache compression method that allocates dimensions to tokens at a more granular level, and MixedDimKV-H, which further integrates head-level importance information. Experiments on long-context benchmarks show that MixedDimKV outperforms prior KV cache compression methods that do not rely on head-level importance profiling. When equipped with the same head-level importance information, MixedDimKV-H consistently outperforms HeadKV. Notably, our approach achieves comparable performance to full attention on LongBench with only 6.25% of the KV cache. Furthermore, in the Needle-in-a-Haystack test, our solution maintains 100% accuracy at a 50K context length while using as little as 0.26% of the cache. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_20616 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression Miao, Ruijie Wang, Zhiming Li, Wang Wu, Shiwei Liu, Shufan Jiang, Yanbing Yang, Tong Machine Learning Key-value (KV) caching is widely used to accelerate transformer inference, but its memory cost grows linearly with input length, limiting long-context deployment. Existing token eviction methods reduce memory by discarding less important tokens, which can be viewed as a coarse form of dimensionality reduction that assigns each token either zero or full dimension. We propose MixedDimKV, a mixed-dimension KV cache compression method that allocates dimensions to tokens at a more granular level, and MixedDimKV-H, which further integrates head-level importance information. Experiments on long-context benchmarks show that MixedDimKV outperforms prior KV cache compression methods that do not rely on head-level importance profiling. When equipped with the same head-level importance information, MixedDimKV-H consistently outperforms HeadKV. Notably, our approach achieves comparable performance to full attention on LongBench with only 6.25% of the KV cache. Furthermore, in the Needle-in-a-Haystack test, our solution maintains 100% accuracy at a 50K context length while using as little as 0.26% of the cache. |
| title | Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2603.20616 |