KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911446955196416 |
|---|---|
| author | Wang, Jiahao Han, Jinbo Wei, Xingda Shen, Sijie Zhang, Dingyan Fang, Chenguang Chen, Rong Yu, Wenyuan Chen, Haibo |
| author_facet | Wang, Jiahao Han, Jinbo Wei, Xingda Shen, Sijie Zhang, Dingyan Fang, Chenguang Chen, Rong Yu, Wenyuan Chen, Haibo |
| contents | Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV\$) after processing each request substantially improves serving throughput and latency. However, there is limited understanding of how LLM serving benefits from KV\$ caching, where system design decisions like cache eviction policies are highly workload-dependent. In this paper, we present the first systematic characterization of the KV\$ workload patterns from one of the leading LLM service providers. We draw observations that were not covered by previous studies focusing on synthetic workloads, including: KV\$ reuses are skewed across requests, where reuses between single-turn requests are equally important as multi-turn requests; the reuse time and probability are diverse considering all requests, but for a specific request category, the pattern tends to be predictable; and the overall cache size required for an ideal cache hit ratio is moderate. Based on the characterization, we further propose a workload-aware cache eviction policy that improves the serving performance under real-world traces, especially with limited cache capacity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02634 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider Wang, Jiahao Han, Jinbo Wei, Xingda Shen, Sijie Zhang, Dingyan Fang, Chenguang Chen, Rong Yu, Wenyuan Chen, Haibo Distributed, Parallel, and Cluster Computing Artificial Intelligence Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV\$) after processing each request substantially improves serving throughput and latency. However, there is limited understanding of how LLM serving benefits from KV\$ caching, where system design decisions like cache eviction policies are highly workload-dependent. In this paper, we present the first systematic characterization of the KV\$ workload patterns from one of the leading LLM service providers. We draw observations that were not covered by previous studies focusing on synthetic workloads, including: KV\$ reuses are skewed across requests, where reuses between single-turn requests are equally important as multi-turn requests; the reuse time and probability are diverse considering all requests, but for a specific request category, the pattern tends to be predictable; and the overall cache size required for an ideal cache hit ratio is moderate. Based on the characterization, we further propose a workload-aware cache eviction policy that improves the serving performance under real-world traces, especially with limited cache capacity. |
| title | KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2506.02634 |