PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912900519559168 |
|---|---|
| author | Woo, Sunghyeon Kim, Hoseung Shim, Sunghwan Jo, Minjung Jeong, Hyunjoon Lee, Jeongtae Kim, Joonghoon Lee, Sungjae Park, Baeseong Kwon, Se Jung Lee, Dongsoo |
| author_facet | Woo, Sunghyeon Kim, Hoseung Shim, Sunghwan Jo, Minjung Jeong, Hyunjoon Lee, Jeongtae Kim, Joonghoon Lee, Sungjae Park, Baeseong Kwon, Se Jung Lee, Dongsoo |
| contents | Multi-agent systems increasingly orchestrate multiple specialized language models to solve complex real-world problems, often invoking them over a shared context. This execution pattern repeatedly processes the same prompt prefix across models. Consequently, each model redundantly executes the prefill stage and maintains its own key-value (KV) cache, increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks. Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs, but disaggregation does not fundamentally eliminate inter-model redundancy in computation and KV storage for the same prompt. To address this issue, we propose PrefillShare, a novel algorithm that enables sharing the prefill stage across multiple models in a disaggregated setting. PrefillShare factorizes the model into prefill and decode modules, freezes the prefill module, and fine-tunes only the decode module. This design allows multiple task-specific models to share a prefill module and the KV cache generated for the same prompt. We further introduce a routing mechanism that enables effective prefill sharing across heterogeneous models in a vLLM-based disaggregated system. PrefillShare not only matches full fine-tuning accuracy on a broad range of tasks and models, but also delivers 4.5x lower p95 latency and 3.9x higher throughput in multi-model agent workloads. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_12029 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving Woo, Sunghyeon Kim, Hoseung Shim, Sunghwan Jo, Minjung Jeong, Hyunjoon Lee, Jeongtae Kim, Joonghoon Lee, Sungjae Park, Baeseong Kwon, Se Jung Lee, Dongsoo Machine Learning Distributed, Parallel, and Cluster Computing Multi-agent systems increasingly orchestrate multiple specialized language models to solve complex real-world problems, often invoking them over a shared context. This execution pattern repeatedly processes the same prompt prefix across models. Consequently, each model redundantly executes the prefill stage and maintains its own key-value (KV) cache, increasing aggregate prefill load and worsening tail latency by intensifying prefill-decode interference in existing LLM serving stacks. Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs, but disaggregation does not fundamentally eliminate inter-model redundancy in computation and KV storage for the same prompt. To address this issue, we propose PrefillShare, a novel algorithm that enables sharing the prefill stage across multiple models in a disaggregated setting. PrefillShare factorizes the model into prefill and decode modules, freezes the prefill module, and fine-tunes only the decode module. This design allows multiple task-specific models to share a prefill module and the KV cache generated for the same prompt. We further introduce a routing mechanism that enables effective prefill sharing across heterogeneous models in a vLLM-based disaggregated system. PrefillShare not only matches full fine-tuning accuracy on a broad range of tasks and models, but also delivers 4.5x lower p95 latency and 3.9x higher throughput in multi-model agent workloads. |
| title | PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2602.12029 |