SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915253326970880 |
|---|---|
| author | Zhao, Yihao Chen, Jiadun Sun, Peng Li, Lei Liu, Xuanzhe Jin, Xin |
| author_facet | Zhao, Yihao Chen, Jiadun Sun, Peng Li, Lei Liu, Xuanzhe Jin, Xin |
| contents | Large language models (LLMs) with different architectures and sizes have been developed. Serving each LLM with dedicated GPUs leads to resource waste and service inefficiency due to the varying demand of LLM requests. A common practice is to share multiple LLMs. However, existing sharing systems either do not consider the autoregressive pattern of LLM services, or only focus on improving the throughput, which impairs the sharing performance, especially the serving latency. We present SeaLLM, which enables service-aware and latency-optimized LLM sharing. SeaLLM improves the overall sharing performance by (1) a latency-optimized scheduling algorithm utilizing the characteristics of LLM services, (2) a placement algorithm to determine the placement plan and an adaptive replacement algorithm to decide the replacement interval, and (3) a unified key-value cache to share GPU memory among LLM services efficiently. Our evaluation under real-world traces and LLM services demonstrates that SeaLLM improves the normalized latency by up to $13.60\times$, the tail latency by up to $18.69\times$, and the SLO attainment by up to $3.64\times$ compared to existing solutions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15720 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference Zhao, Yihao Chen, Jiadun Sun, Peng Li, Lei Liu, Xuanzhe Jin, Xin Distributed, Parallel, and Cluster Computing Large language models (LLMs) with different architectures and sizes have been developed. Serving each LLM with dedicated GPUs leads to resource waste and service inefficiency due to the varying demand of LLM requests. A common practice is to share multiple LLMs. However, existing sharing systems either do not consider the autoregressive pattern of LLM services, or only focus on improving the throughput, which impairs the sharing performance, especially the serving latency. We present SeaLLM, which enables service-aware and latency-optimized LLM sharing. SeaLLM improves the overall sharing performance by (1) a latency-optimized scheduling algorithm utilizing the characteristics of LLM services, (2) a placement algorithm to determine the placement plan and an adaptive replacement algorithm to decide the replacement interval, and (3) a unified key-value cache to share GPU memory among LLM services efficiently. Our evaluation under real-world traces and LLM services demonstrates that SeaLLM improves the normalized latency by up to $13.60\times$, the tail latency by up to $18.69\times$, and the SLO attainment by up to $3.64\times$ compared to existing solutions. |
| title | SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2504.15720 |