IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910254299611136 |
|---|---|
| author | Yang, Xintong Gu, Hao Xu, Binxing Li, Lujun Liu, Bei Liu, Jiacheng Zhu, Qiyuan Han, Sirui Guo, Yike |
| author_facet | Yang, Xintong Gu, Hao Xu, Binxing Li, Lujun Liu, Bei Liu, Jiacheng Zhu, Qiyuan Han, Sirui Guo, Yike |
| contents | Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference. A practical remedy is to evict less important KV entries; however, existing eviction policies are largely heuristic and struggle to capture the rich, input-dependent distribution of token importance. In this work, we introduce a learnable indexer that predicts KV importance, enabling more accurate retention of critical tokens. Meanwhile, naively evicting tokens permanently discards their information, leading to irreversible forgetting and degraded retrieval over long ranges. To address this, we propose a lightweight latent memory module that compresses evicted tokens into a compact, online-updated state and provides residual readouts to compensate for the attention contributions lost through KV eviction. Collectively, our method enables accurate long-context inference under a bounded KV budget, delivering consistent improvements on RULER (4K/16K) across Qwen, Mistral, and Llama models (up to 25 points under aggressive eviction), markedly more stable Needle-in-a-Haystack retrieval, and superior LongBench scores and compression curves compared to existing eviction policies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25475 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference Yang, Xintong Gu, Hao Xu, Binxing Li, Lujun Liu, Bei Liu, Jiacheng Zhu, Qiyuan Han, Sirui Guo, Yike Computation and Language Artificial Intelligence Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference. A practical remedy is to evict less important KV entries; however, existing eviction policies are largely heuristic and struggle to capture the rich, input-dependent distribution of token importance. In this work, we introduce a learnable indexer that predicts KV importance, enabling more accurate retention of critical tokens. Meanwhile, naively evicting tokens permanently discards their information, leading to irreversible forgetting and degraded retrieval over long ranges. To address this, we propose a lightweight latent memory module that compresses evicted tokens into a compact, online-updated state and provides residual readouts to compensate for the attention contributions lost through KV eviction. Collectively, our method enables accurate long-context inference under a bounded KV budget, delivering consistent improvements on RULER (4K/16K) across Qwen, Mistral, and Llama models (up to 25 points under aggressive eviction), markedly more stable Needle-in-a-Haystack retrieval, and superior LongBench scores and compression curves compared to existing eviction policies. |
| title | IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.25475 |