Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913796334813184 |
|---|---|
| author | Kim, Kihyun Kim, Jinwoo Chung, Hyunsun Cha, Myung-Hoon Kim, Hong-Yeon Kim, Youngjae |
| author_facet | Kim, Kihyun Kim, Jinwoo Chung, Hyunsun Cha, Myung-Hoon Kim, Hong-Yeon Kim, Youngjae |
| contents | LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS is a major burden. This paper proposes InferSave, a cost-efficient VM selection framework for cloud based LLM inference. InferSave optimizes KV cache offloading based on Service Level Objectives (SLOs) and workload charac teristics, estimating GPU memory needs, and recommending cost-effective VM instances. Additionally, the Compute Time Calibration Function (CTCF) improves instance selection accuracy by adjusting for discrepancies between theoretical and actual GPU performance. Experiments on AWS GPU instances show that selecting lower-cost instances without KV cache offloading improves cost efficiency by up to 73.7% for online workloads, while KV cache offloading saves up to 20.19% for offline workloads. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_11816 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading Kim, Kihyun Kim, Jinwoo Chung, Hyunsun Cha, Myung-Hoon Kim, Hong-Yeon Kim, Youngjae Machine Learning Distributed, Parallel, and Cluster Computing LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS is a major burden. This paper proposes InferSave, a cost-efficient VM selection framework for cloud based LLM inference. InferSave optimizes KV cache offloading based on Service Level Objectives (SLOs) and workload charac teristics, estimating GPU memory needs, and recommending cost-effective VM instances. Additionally, the Compute Time Calibration Function (CTCF) improves instance selection accuracy by adjusting for discrepancies between theoretical and actual GPU performance. Experiments on AWS GPU instances show that selecting lower-cost instances without KV cache offloading improves cost efficiency by up to 73.7% for online workloads, while KV cache offloading saves up to 20.19% for offline workloads. |
| title | Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2504.11816 |