TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918512348364800 |
|---|---|
| author | Bian, Zhuohang Wu, Feiyang Li, Zhuoran Ma, Teng Zhuo, Youwei |
| author_facet | Bian, Zhuohang Wu, Feiyang Li, Zhuoran Ma, Teng Zhuo, Youwei |
| contents | Large Language Models (LLMs) are increasingly deployed in complex multi-agent applications that rely on external function calls. This workload creates severe performance challenges for the KV Cache: spatial contention leads to the eviction of critical agents' caches and temporal underutilization leaves the cache of agents stalled on long-running function calls idling in GPU memory. We present TokenCake, a KV-Cache-centric serving framework that bridges this gap by co-optimizing scheduling and memory management through an agent-aware design. TokenCake's Temporal Scheduler employs an event-driven, opportunistic policy to proactively offload idle KV Caches during function calls and uses predictive uploading to hide data transfer latency. TokenCake's Spatial Scheduler uses dynamic memory partitioning, guided by a hybrid priority metric combining graph structure and runtime state, to reserve GPU memory for critical-path agents. Our evaluation on representative multi-agent benchmarks shows that TokenCake reduces end-to-end latency by over 47.06% and improves effective GPU memory utilization by up to 16.9% compared to vLLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_18586 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications Bian, Zhuohang Wu, Feiyang Li, Zhuoran Ma, Teng Zhuo, Youwei Distributed, Parallel, and Cluster Computing C.4; D.4.2; I.2.11 Large Language Models (LLMs) are increasingly deployed in complex multi-agent applications that rely on external function calls. This workload creates severe performance challenges for the KV Cache: spatial contention leads to the eviction of critical agents' caches and temporal underutilization leaves the cache of agents stalled on long-running function calls idling in GPU memory. We present TokenCake, a KV-Cache-centric serving framework that bridges this gap by co-optimizing scheduling and memory management through an agent-aware design. TokenCake's Temporal Scheduler employs an event-driven, opportunistic policy to proactively offload idle KV Caches during function calls and uses predictive uploading to hide data transfer latency. TokenCake's Spatial Scheduler uses dynamic memory partitioning, guided by a hybrid priority metric combining graph structure and runtime state, to reserve GPU memory for critical-path agents. Our evaluation on representative multi-agent benchmarks shows that TokenCake reduces end-to-end latency by over 47.06% and improves effective GPU memory utilization by up to 16.9% compared to vLLM. |
| title | TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications |
| topic | Distributed, Parallel, and Cluster Computing C.4; D.4.2; I.2.11 |
| url | https://arxiv.org/abs/2510.18586 |