ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915504785981440 |
|---|---|
| author | Xiang, Xingyu Joshi, Raj Liu, Yuhan Yao, Jiayi Zhao, Chenxingyu Jiang, Junchen Zhou, Yang Kohler, Eddie Yu, Minlan |
| author_facet | Xiang, Xingyu Joshi, Raj Liu, Yuhan Yao, Jiayi Zhao, Chenxingyu Jiang, Junchen Zhou, Yang Kohler, Eddie Yu, Minlan |
| contents | Distributed prefix caching accelerates long-context LLM serving by reusing KV cache entries for common context prefixes. However, KV cache fetches can become a bottleneck when network bandwidth is limited. Compression mitigates the bandwidth issue, but can degrade overall performance when decompression interferes with model computation.
We present ShadowServe, the first SmartNIC-accelerated, interference-free prefix caching system for LLM serving. ShadowServe separates a control plane on the host and a data plane fully offloaded to the SmartNIC, which eliminates interference to both host GPU and CPU. To overcome the SmartNIC's limited compute and memory resources, we design a chunked pipeline that parallelizes data plane operations across the SmartNIC's compute resources, and a minimal-copy memory management scheme that reduces memory pressure on the SmartNIC. Compared to state-of-the-art solutions, ShadowServe achieves up to 2.2x lower loaded time-per-output-token (TPOT), and reduces time-to-first-token (TTFT) by up to 1.38x in low-bandwidth scenarios (<= 20 Gbps), translating to up to 1.35x higher throughput. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16857 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching Xiang, Xingyu Joshi, Raj Liu, Yuhan Yao, Jiayi Zhao, Chenxingyu Jiang, Junchen Zhou, Yang Kohler, Eddie Yu, Minlan Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Distributed prefix caching accelerates long-context LLM serving by reusing KV cache entries for common context prefixes. However, KV cache fetches can become a bottleneck when network bandwidth is limited. Compression mitigates the bandwidth issue, but can degrade overall performance when decompression interferes with model computation. We present ShadowServe, the first SmartNIC-accelerated, interference-free prefix caching system for LLM serving. ShadowServe separates a control plane on the host and a data plane fully offloaded to the SmartNIC, which eliminates interference to both host GPU and CPU. To overcome the SmartNIC's limited compute and memory resources, we design a chunked pipeline that parallelizes data plane operations across the SmartNIC's compute resources, and a minimal-copy memory management scheme that reduces memory pressure on the SmartNIC. Compared to state-of-the-art solutions, ShadowServe achieves up to 2.2x lower loaded time-per-output-token (TPOT), and reduces time-to-first-token (TTFT) by up to 1.38x in low-bandwidth scenarios (<= 20 Gbps), translating to up to 1.35x higher throughput. |
| title | ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2509.16857 |