HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908557434159104 |
|---|---|
| author | Lou, Chiheng Qi, Sheng Jin, Chao Nie, Dapeng Yang, Haoran Ding, Yu Liu, Xuanzhe Jin, Xin |
| author_facet | Lou, Chiheng Qi, Sheng Jin, Chao Nie, Dapeng Yang, Haoran Ding, Yu Liu, Xuanzhe Jin, Xin |
| contents | With the proliferation of large language model (LLM) variants, developers are turning to serverless computing for cost-efficient LLM deployment. However, public cloud providers often struggle to provide performance guarantees for serverless LLM serving due to significant cold start latency caused by substantial model sizes and complex runtime dependencies. To address this problem, we present HydraServe, a serverless LLM serving system designed to minimize cold start latency in public clouds. HydraServe proactively distributes models across servers to quickly fetch them, and overlaps cold-start stages within workers to reduce startup latency. Additionally, HydraServe strategically places workers across GPUs to avoid network contention among cold-start instances. To minimize resource consumption during cold starts, HydraServe further introduces pipeline consolidation that can merge groups of workers into individual serving endpoints. Our comprehensive evaluations under diverse settings demonstrate that HydraServe reduces the cold start latency by 1.7$\times$-- 4.7$\times$ and improves service level objective attainment by 1.43$\times$--1.74$\times$ compared to baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15524 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds Lou, Chiheng Qi, Sheng Jin, Chao Nie, Dapeng Yang, Haoran Ding, Yu Liu, Xuanzhe Jin, Xin Distributed, Parallel, and Cluster Computing With the proliferation of large language model (LLM) variants, developers are turning to serverless computing for cost-efficient LLM deployment. However, public cloud providers often struggle to provide performance guarantees for serverless LLM serving due to significant cold start latency caused by substantial model sizes and complex runtime dependencies. To address this problem, we present HydraServe, a serverless LLM serving system designed to minimize cold start latency in public clouds. HydraServe proactively distributes models across servers to quickly fetch them, and overlaps cold-start stages within workers to reduce startup latency. Additionally, HydraServe strategically places workers across GPUs to avoid network contention among cold-start instances. To minimize resource consumption during cold starts, HydraServe further introduces pipeline consolidation that can merge groups of workers into individual serving endpoints. Our comprehensive evaluations under diverse settings demonstrate that HydraServe reduces the cold start latency by 1.7$\times$-- 4.7$\times$ and improves service level objective attainment by 1.43$\times$--1.74$\times$ compared to baselines. |
| title | HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2502.15524 |