KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910217190506496 |
|---|---|
| author | Liu, Zedong Ma, Xinyang Luo, Dejun Zhao, Hairui Lu, Bing Huang, Wenjing Gu, Yida Liu, Xingchen Wei, Zheng Liu, Jinyang Tao, Dingwen Tan, Guangming |
| author_facet | Liu, Zedong Ma, Xinyang Luo, Dejun Zhao, Hairui Lu, Bing Huang, Wenjing Gu, Yida Liu, Xingchen Wei, Zheng Liu, Jinyang Tao, Dingwen Tan, Guangming |
| contents | LLMs are widely adopted in production, pushing inference systems to their limits. Disaggregated LLM serving (e.g., PD separation and KV state disaggregation) improves scalability and cost efficiency, but it also turns KV into an explicit payload crossing network and storage boundaries, making KV a dominant end-to-end bottleneck. Existing KV compression are typically static runtime configurations, despite production service context varies over time in workload mix, bandwidth, and SLO/quality budgets. As a result, a fixed choice can be suboptimal or even increase latency. We present \emph{KVServe}, the first service-aware and adaptive KV communication compression framework for disaggregated LLM serving: KVServe (1) unifies KV compression into a modular strategy space with new components and cross-method recomposition; (2) introduces Bayesian Profiling Engine that efficiently searches this space and distills a 3D Pareto candidate set, reducing $50\times$ offline search overhead; and (3) deploys a Service-Aware Online Controller that combines an analytical latency model with a lightweight bandit to select profiles under constraints and correct offline-to-online mismatch. Integrated into vLLM and evaluated across datasets, models, GPUs and networks, KVServe achieves up to $9.13\times$ JCT speedup in PD-separated serving and up to $32.8\times$ TTFT reduction in KV-disaggregated serving. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_13734 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving Liu, Zedong Ma, Xinyang Luo, Dejun Zhao, Hairui Lu, Bing Huang, Wenjing Gu, Yida Liu, Xingchen Wei, Zheng Liu, Jinyang Tao, Dingwen Tan, Guangming Distributed, Parallel, and Cluster Computing Artificial Intelligence Networking and Internet Architecture LLMs are widely adopted in production, pushing inference systems to their limits. Disaggregated LLM serving (e.g., PD separation and KV state disaggregation) improves scalability and cost efficiency, but it also turns KV into an explicit payload crossing network and storage boundaries, making KV a dominant end-to-end bottleneck. Existing KV compression are typically static runtime configurations, despite production service context varies over time in workload mix, bandwidth, and SLO/quality budgets. As a result, a fixed choice can be suboptimal or even increase latency. We present \emph{KVServe}, the first service-aware and adaptive KV communication compression framework for disaggregated LLM serving: KVServe (1) unifies KV compression into a modular strategy space with new components and cross-method recomposition; (2) introduces Bayesian Profiling Engine that efficiently searches this space and distills a 3D Pareto candidate set, reducing $50\times$ offline search overhead; and (3) deploys a Service-Aware Online Controller that combines an analytical latency model with a lightweight bandit to select profiles under constraints and correct offline-to-online mismatch. Integrated into vLLM and evaluated across datasets, models, GPUs and networks, KVServe achieves up to $9.13\times$ JCT speedup in PD-separated serving and up to $32.8\times$ TTFT reduction in KV-disaggregated serving. |
| title | KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Networking and Internet Architecture |
| url | https://arxiv.org/abs/2605.13734 |