A Simple Plug-in for Improving Eviction-Based KV Cache Compression
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916038408404992 |
|---|---|
| author | Lin, Yuping Ding, Jiayuan Xing, Yue He, Pengfei Tang, Jiliang Mukherjee, Subhabrata |
| author_facet | Lin, Yuping Ding, Jiayuan Xing, Yue He, Pengfei Tang, Jiliang Mukherjee, Subhabrata |
| contents | KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction or representation approximation, which may underutilize tokens that are not critical for exact retention but are still reconstructable. We present VECTOR, a plug-and-play augmentation for eviction-based pipelines that introduces three-way token routing: retention, approximation, and eviction. VECTOR combines an importance signal from the base scorer with a reconstructability signal from an offline-calibrated regression-based value estimation. By leveraging reconstructability, VECTOR recovers useful value information that would otherwise be irreversibly lost under binary eviction, while preserving key vectors for attention routing stability. Experimental results show that VECTOR improves quality-memory trade-offs under medium-to-high compression, with especially clear gains in stricter budget regimes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_23258 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Simple Plug-in for Improving Eviction-Based KV Cache Compression Lin, Yuping Ding, Jiayuan Xing, Yue He, Pengfei Tang, Jiliang Mukherjee, Subhabrata Machine Learning KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction or representation approximation, which may underutilize tokens that are not critical for exact retention but are still reconstructable. We present VECTOR, a plug-and-play augmentation for eviction-based pipelines that introduces three-way token routing: retention, approximation, and eviction. VECTOR combines an importance signal from the base scorer with a reconstructability signal from an offline-calibrated regression-based value estimation. By leveraging reconstructability, VECTOR recovers useful value information that would otherwise be irreversibly lost under binary eviction, while preserving key vectors for attention routing stability. Experimental results show that VECTOR improves quality-memory trade-offs under medium-to-high compression, with especially clear gains in stricter budget regimes. |
| title | A Simple Plug-in for Improving Eviction-Based KV Cache Compression |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.23258 |