DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916145810898944 |
|---|---|
| author | Strati, Foteini Mcallister, Sara Phanishayee, Amar Tarnawski, Jakub Klimovic, Ana |
| author_facet | Strati, Foteini Mcallister, Sara Phanishayee, Amar Tarnawski, Jakub Klimovic, Ana |
| contents | Distributed LLM serving is costly and often underutilizes hardware accelerators due to three key challenges: bubbles in pipeline-parallel deployments caused by the bimodal latency of prompt and token processing, GPU memory overprovisioning, and long recovery times in case of failures. In this paper, we propose DéjàVu, a system to address all these challenges using a versatile and efficient KV cache streaming library (DéjàVuLib). Using DéjàVuLib, we propose and implement efficient prompt-token disaggregation to reduce pipeline bubbles, microbatch swapping for efficient GPU memory management, and state replication for fault-tolerance. We highlight the efficacy of these solutions on a range of large models across cloud deployments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01876 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving Strati, Foteini Mcallister, Sara Phanishayee, Amar Tarnawski, Jakub Klimovic, Ana Distributed, Parallel, and Cluster Computing Distributed LLM serving is costly and often underutilizes hardware accelerators due to three key challenges: bubbles in pipeline-parallel deployments caused by the bimodal latency of prompt and token processing, GPU memory overprovisioning, and long recovery times in case of failures. In this paper, we propose DéjàVu, a system to address all these challenges using a versatile and efficient KV cache streaming library (DéjàVuLib). Using DéjàVuLib, we propose and implement efficient prompt-token disaggregation to reduce pipeline bubbles, microbatch swapping for efficient GPU memory management, and state replication for fault-tolerance. We highlight the efficacy of these solutions on a range of large models across cloud deployments. |
| title | DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2403.01876 |