TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911565310066688 |
|---|---|
| author | Bian, Zhuohang Wu, Feiyang Zhang, Chengrui Dong, Hangcheng Liang, Yun Zhuo, Youwei |
| author_facet | Bian, Zhuohang Wu, Feiyang Zhang, Chengrui Dong, Hangcheng Liang, Yun Zhuo, Youwei |
| contents | Multi-agent LLM applications organize execution in synchronized rounds where a central scheduler gathers outputs from all agents and redistributes the combined context. This All-Gather communication pattern creates massive KV Cache redundancy, because every agent's prompt contains the same shared output blocks, yet existing reuse methods fail to exploit it efficiently. We present TokenDance, a system that scales the number of concurrent agents by exploiting the All-Gather pattern for collective KV Cache sharing. TokenDance's KV Collector performs KV Cache reuse over the full round in one collective step, so the cost of reusing a shared block is paid once regardless of agent count. Its Diff-Aware Storage encodes sibling caches as block-sparse diffs against a single master copy, achieving 11-17x compression on representative workloads. Evaluation on GenerativeAgents and AgentSociety shows that TokenDance supports up to 2.7x more concurrent agents than vLLM with prefix caching under SLO requirement, reduces per-agent KV Cache storage by up to 17.5x, and achieves up to 1.9x prefill speedup over per-request position-independent caching. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_03143 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing Bian, Zhuohang Wu, Feiyang Zhang, Chengrui Dong, Hangcheng Liang, Yun Zhuo, Youwei Distributed, Parallel, and Cluster Computing Multi-agent LLM applications organize execution in synchronized rounds where a central scheduler gathers outputs from all agents and redistributes the combined context. This All-Gather communication pattern creates massive KV Cache redundancy, because every agent's prompt contains the same shared output blocks, yet existing reuse methods fail to exploit it efficiently. We present TokenDance, a system that scales the number of concurrent agents by exploiting the All-Gather pattern for collective KV Cache sharing. TokenDance's KV Collector performs KV Cache reuse over the full round in one collective step, so the cost of reusing a shared block is paid once regardless of agent count. Its Diff-Aware Storage encodes sibling caches as block-sparse diffs against a single master copy, achieving 11-17x compression on representative workloads. Evaluation on GenerativeAgents and AgentSociety shows that TokenDance supports up to 2.7x more concurrent agents than vLLM with prefix caching under SLO requirement, reduces per-agent KV Cache storage by up to 17.5x, and achieves up to 1.9x prefill speedup over per-request position-independent caching. |
| title | TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2604.03143 |