CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915881202745344 |
|---|---|
| author | Wang, Weiye Chen, Chen Zhang, Junxue Wang, Zhusheng Yuan, Hui Guan, Zixuan Zheng, Xiaolong Weng, Qizhen Chen, Yin Guo, Minyi |
| author_facet | Wang, Weiye Chen, Chen Zhang, Junxue Wang, Zhusheng Yuan, Hui Guan, Zixuan Zheng, Xiaolong Weng, Qizhen Chen, Yin Guo, Minyi |
| contents | Distributed prefix caching has become a core technique for efficient LLM serving. However, for long-context requests with high cache hit ratios, retrieving reusable KVCache blocks from remote servers has emerged as a new performance bottleneck. Such network-intensive LLM inference is expected to become increasingly common as agentic AI workloads continue to grow. However, existing LLM inference engines remain largely compute-centric: they treat KVCache loading as a subordinate phase to GPU execution and often fail to account for its delay explicitly during scheduling.
We present CALVO, an LLM serving engine that treats KVCache loading as a first-class concern. CALVO decouples KVCache loading and GPU computation into independently managed, asynchronously progressing stages, enabling better utilization of network, PCIe, and computation resources. In addition, CALVO incorporates KVCache loading delay as an explicit component of per-request service cost, leading to more accurate scheduling decisions. Experiments on a real testbed with diverse long-context workloads show that CALVO substantially improves the efficiency of network-intensive LLM inference, achieving up to 61.67% higher SLO attainment than the baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_21257 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands Wang, Weiye Chen, Chen Zhang, Junxue Wang, Zhusheng Yuan, Hui Guan, Zixuan Zheng, Xiaolong Weng, Qizhen Chen, Yin Guo, Minyi Distributed, Parallel, and Cluster Computing Distributed prefix caching has become a core technique for efficient LLM serving. However, for long-context requests with high cache hit ratios, retrieving reusable KVCache blocks from remote servers has emerged as a new performance bottleneck. Such network-intensive LLM inference is expected to become increasingly common as agentic AI workloads continue to grow. However, existing LLM inference engines remain largely compute-centric: they treat KVCache loading as a subordinate phase to GPU execution and often fail to account for its delay explicitly during scheduling. We present CALVO, an LLM serving engine that treats KVCache loading as a first-class concern. CALVO decouples KVCache loading and GPU computation into independently managed, asynchronously progressing stages, enabling better utilization of network, PCIe, and computation resources. In addition, CALVO incorporates KVCache loading delay as an explicit component of per-request service cost, leading to more accurate scheduling decisions. Experiments on a real testbed with diverse long-context workloads show that CALVO substantially improves the efficiency of network-intensive LLM inference, achieving up to 61.67% higher SLO attainment than the baseline. |
| title | CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2603.21257 |