Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915622011535360 |
|---|---|
| author | Wang, Yixuan Ji, Shiyu Liu, Yijun Xu, Yuzhuang Xu, Yang Zhu, Qingfu Che, Wanxiang |
| author_facet | Wang, Yixuan Ji, Shiyu Liu, Yijun Xu, Yuzhuang Xu, Yang Zhu, Qingfu Che, Wanxiang |
| contents | Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challenges for efficient deployment. Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries, especially under tight memory budgets. In this paper, we propose Lookahead Q-Cache (LAQ), a novel eviction framework that generates low-cost pseudo lookahead queries to better approximate the true decoding-stage queries. By using these lookahead queries as the observation window for importance estimation, LAQ achieves more consistent and accurate KV cache eviction aligned with real inference scenarios. Experimental results on LongBench and Needle-in-a-Haystack benchmarks show that LAQ outperforms existing methods across various budget levels, achieving a 1 $\sim$ 4 point improvement on LongBench under limited cache budget. Moreover, LAQ is complementary to existing approaches and can be flexibly combined to yield further improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20334 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query Wang, Yixuan Ji, Shiyu Liu, Yijun Xu, Yuzhuang Xu, Yang Zhu, Qingfu Che, Wanxiang Computation and Language Artificial Intelligence Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challenges for efficient deployment. Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries, especially under tight memory budgets. In this paper, we propose Lookahead Q-Cache (LAQ), a novel eviction framework that generates low-cost pseudo lookahead queries to better approximate the true decoding-stage queries. By using these lookahead queries as the observation window for importance estimation, LAQ achieves more consistent and accurate KV cache eviction aligned with real inference scenarios. Experimental results on LongBench and Needle-in-a-Haystack benchmarks show that LAQ outperforms existing methods across various budget levels, achieving a 1 $\sim$ 4 point improvement on LongBench under limited cache budget. Moreover, LAQ is complementary to existing approaches and can be flexibly combined to yield further improvements. |
| title | Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.20334 |