λScale: Enabling Fast Scaling for Serverless Large Language Model Inference
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910042668662784 |
|---|---|
| author | Yu, Minchen Yang, Rui Jia, Chaobo Su, Zhaoyuan Yao, Sheng Lan, Tingfeng Yang, Yuchen Wang, Zirui Cheng, Yue Wang, Wei Wang, Ao Chen, Ruichuan |
| author_facet | Yu, Minchen Yang, Rui Jia, Chaobo Su, Zhaoyuan Yao, Sheng Lan, Tingfeng Yang, Yuchen Wang, Zirui Cheng, Yue Wang, Wei Wang, Ao Chen, Ruichuan |
| contents | Serverless computing has emerged as a compelling solution for cloud-based model inference. However, as modern large language models (LLMs) continue to grow in size, existing serverless platforms often face substantial model startup overhead. This poses a significant challenge in efficiently scaling model instances to accommodate dynamic, bursty workloads commonly observed in real-world inference services. In this paper, we introduce λScale, an efficient serverless inference system to achieve fast model scaling. The key idea behind λScale is to leverage high-speed RDMA networks between GPU nodes for fast model multicast, while enabling distributed inference execution during model transmission -- referred to as "execute-while-load". λScale proposes an efficient model scaling scheme, λPipe, which supports adaptive model multicast and dynamically constructs execution pipelines across receiving nodes for collaborative, distributed inference. Additionally, λScale supports efficient model management across GPU and host memory, allowing fast scaling for models across different storage tiers. Evaluation results show that λScale enables fast model scaling and effectively handles load spikes, achieving up to 5x tail-latency improvement and 31.3% cost reduction compared to state-of-the-art solutions on real-world LLM inference traces. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_09922 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | λScale: Enabling Fast Scaling for Serverless Large Language Model Inference Yu, Minchen Yang, Rui Jia, Chaobo Su, Zhaoyuan Yao, Sheng Lan, Tingfeng Yang, Yuchen Wang, Zirui Cheng, Yue Wang, Wei Wang, Ao Chen, Ruichuan Distributed, Parallel, and Cluster Computing Serverless computing has emerged as a compelling solution for cloud-based model inference. However, as modern large language models (LLMs) continue to grow in size, existing serverless platforms often face substantial model startup overhead. This poses a significant challenge in efficiently scaling model instances to accommodate dynamic, bursty workloads commonly observed in real-world inference services. In this paper, we introduce λScale, an efficient serverless inference system to achieve fast model scaling. The key idea behind λScale is to leverage high-speed RDMA networks between GPU nodes for fast model multicast, while enabling distributed inference execution during model transmission -- referred to as "execute-while-load". λScale proposes an efficient model scaling scheme, λPipe, which supports adaptive model multicast and dynamically constructs execution pipelines across receiving nodes for collaborative, distributed inference. Additionally, λScale supports efficient model management across GPU and host memory, allowing fast scaling for models across different storage tiers. Evaluation results show that λScale enables fast model scaling and effectively handles load spikes, achieving up to 5x tail-latency improvement and 31.3% cost reduction compared to state-of-the-art solutions on real-world LLM inference traces. |
| title | λScale: Enabling Fast Scaling for Serverless Large Language Model Inference |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2502.09922 |