AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917205984149504 |
|---|---|
| author | Feng, Shaoting Li, Hanchen Du, Kuntai Gu, Zhuohan Liu, Yuhan Yao, Jiayi Ray, Siddhant Shen, Samuel Cheng, Yihua Ananthanarayanan, Ganesh Jiang, Junchen |
| author_facet | Feng, Shaoting Li, Hanchen Du, Kuntai Gu, Zhuohan Liu, Yuhan Yao, Jiayi Ray, Siddhant Shen, Samuel Cheng, Yihua Ananthanarayanan, Ganesh Jiang, Junchen |
| contents | Large language model (LLM) applications often reuse previously processed context, such as chat history and documents, which introduces significant redundant computation. Existing LLM serving systems address such redundant computation by storing the KV caches of processed context and loading the corresponding KV cache when a new request reuses the context. Further, as these LLM applications scale, the total size of KV caches becomes excessively large and requires both DRAM and SSD for full storage.
However, prior work that stores KV caches in DRAM and SSD suffers from high loading delays, as most KV cache hits come from SSD, which is slow to load. To increase the KV cache hit rate on DRAM, we identify lossy KV cache compression as a promising approach. We design a lossy compression system that decides the compression algorithm, compression rate and device placement for each KV cache entry to maximise DRAM hits and minimise loading delay without significantly degrading generation quality. Compared to various static compression baselines across three tasks, our system AdaptCache achieves 1.43--2.4 x delay savings at the same quality and 6--55% quality improvements at the same delay. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_00105 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving Feng, Shaoting Li, Hanchen Du, Kuntai Gu, Zhuohan Liu, Yuhan Yao, Jiayi Ray, Siddhant Shen, Samuel Cheng, Yihua Ananthanarayanan, Ganesh Jiang, Junchen Operating Systems Artificial Intelligence Machine Learning Large language model (LLM) applications often reuse previously processed context, such as chat history and documents, which introduces significant redundant computation. Existing LLM serving systems address such redundant computation by storing the KV caches of processed context and loading the corresponding KV cache when a new request reuses the context. Further, as these LLM applications scale, the total size of KV caches becomes excessively large and requires both DRAM and SSD for full storage. However, prior work that stores KV caches in DRAM and SSD suffers from high loading delays, as most KV cache hits come from SSD, which is slow to load. To increase the KV cache hit rate on DRAM, we identify lossy KV cache compression as a promising approach. We design a lossy compression system that decides the compression algorithm, compression rate and device placement for each KV cache entry to maximise DRAM hits and minimise loading delay without significantly degrading generation quality. Compared to various static compression baselines across three tasks, our system AdaptCache achieves 1.43--2.4 x delay savings at the same quality and 6--55% quality improvements at the same delay. |
| title | AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving |
| topic | Operating Systems Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2509.00105 |