EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915680953040896 |
|---|---|
| author | Feng, Shaoting Liu, Yuhan Li, Hanchen Chen, Xiaokun Shen, Samuel Du, Kuntai Gu, Zhuohan Zhang, Rui Huang, Yuyang Cheng, Yihua Yao, Jiayi Zhang, Qizheng Ananthanarayanan, Ganesh Jiang, Junchen |
| author_facet | Feng, Shaoting Liu, Yuhan Li, Hanchen Chen, Xiaokun Shen, Samuel Du, Kuntai Gu, Zhuohan Zhang, Rui Huang, Yuyang Cheng, Yihua Yao, Jiayi Zhang, Qizheng Ananthanarayanan, Ganesh Jiang, Junchen |
| contents | Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity, so prior work has proposed to either evict KV cache to lower-tier storage devices, or compress KV cache so that more KV cache can be fit in the fast memory. However, prior work misses an important opportunity: jointly optimizing the eviction and compression decisions across all KV caches to minimize average generation latency without hurting quality.
We propose EVICPRESS, a KV-cache management system that applies lossy compression and adaptive eviction to KV cache across multiple storage tiers. Specifically, for each KV cache of a context, EVICPRESS considers the effect of compression and eviction of the KV cache on the average generation quality and delay across all contexts as a whole. To achieve this, EVICPRESS proposes a unified utility function that quantifies the effect of quality and delay of the lossy compression or eviction. To this end, EVICPRESS's profiling module periodically updates the utility function scores on all possible eviction-compression configurations for all contexts and places KV caches using a fast heuristic to rearrange KV caches on all storage tiers, with the goal of maximizing the utility function scores on each storage tier. Compared to the baselines that evict KV cache or compress KV cache, EVICPRESS achieves higher KV-cache hit rates on fast devices, i.e., lower delay, while preserving high generation quality by applying conservative compression to contexts that are sensitive to compression errors. Evaluation on 12 datasets and 5 models demonstrates that EVICPRESS achieves up to 2.19x faster time-to-first-token (TTFT) at equivalent generation quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_14946 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving Feng, Shaoting Liu, Yuhan Li, Hanchen Chen, Xiaokun Shen, Samuel Du, Kuntai Gu, Zhuohan Zhang, Rui Huang, Yuyang Cheng, Yihua Yao, Jiayi Zhang, Qizheng Ananthanarayanan, Ganesh Jiang, Junchen Operating Systems Artificial Intelligence Machine Learning Reusing KV cache is essential for high efficiency of Large Language Model (LLM) inference systems. With more LLM users, the KV cache footprint can easily exceed GPU memory capacity, so prior work has proposed to either evict KV cache to lower-tier storage devices, or compress KV cache so that more KV cache can be fit in the fast memory. However, prior work misses an important opportunity: jointly optimizing the eviction and compression decisions across all KV caches to minimize average generation latency without hurting quality. We propose EVICPRESS, a KV-cache management system that applies lossy compression and adaptive eviction to KV cache across multiple storage tiers. Specifically, for each KV cache of a context, EVICPRESS considers the effect of compression and eviction of the KV cache on the average generation quality and delay across all contexts as a whole. To achieve this, EVICPRESS proposes a unified utility function that quantifies the effect of quality and delay of the lossy compression or eviction. To this end, EVICPRESS's profiling module periodically updates the utility function scores on all possible eviction-compression configurations for all contexts and places KV caches using a fast heuristic to rearrange KV caches on all storage tiers, with the goal of maximizing the utility function scores on each storage tier. Compared to the baselines that evict KV cache or compress KV cache, EVICPRESS achieves higher KV-cache hit rates on fast devices, i.e., lower delay, while preserving high generation quality by applying conservative compression to contexts that are sensitive to compression errors. Evaluation on 12 datasets and 5 models demonstrates that EVICPRESS achieves up to 2.19x faster time-to-first-token (TTFT) at equivalent generation quality. |
| title | EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving |
| topic | Operating Systems Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2512.14946 |