Memory Offloading for Large Language Model Inference with Latency SLO Guarantees
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915148030017536 |
|---|---|
| author | Ma, Chenxiang Ye, Zhisheng Zhao, Hanyu Yang, Zehua Fu, Tianhao Han, Jiaxun Zhang, Jie Luo, Yingwei Wang, Xiaolin Wang, Zhenlin Li, Yong Zhou, Diyu |
| author_facet | Ma, Chenxiang Ye, Zhisheng Zhao, Hanyu Yang, Zehua Fu, Tianhao Han, Jiaxun Zhang, Jie Luo, Yingwei Wang, Xiaolin Wang, Zhenlin Li, Yong Zhou, Diyu |
| contents | Offloading large language models (LLMs) state to host memory during inference promises to reduce operational costs by supporting larger models, longer inputs, and larger batch sizes. However, the design of existing memory offloading mechanisms does not take latency service-level objectives (SLOs) into consideration. As a result, they either lead to frequent SLO violations or underutilize host memory, thereby incurring economic loss and thus defeating the purpose of memory offloading.
This paper presents Select-N, a latency-SLO-aware memory offloading system for LLM serving. A key challenge in designing Select-N is to reconcile the tension between meeting SLOs and maximizing host memory usage. Select-N overcomes it by exploiting a unique characteristic of modern LLMs: during serving, the computation time of each decoder layer is deterministic. Leveraging this, Select-N introduces offloading interval, an internal tunable knob that captures the tradeoff between SLOs and host memory usage, thereby reducing the aforementioned challenge to pick an optimal offloading interval. With that, Select-N proposes a two-stage approach to automatically pick the offloading interval. The first stage is offline that generates the range of optimal offloading interval, while the second stage adjusts offloading interval at the granularity of inference iteration based on runtime hardware status. Our evaluation shows that Select-N consistently meets SLOs and improves the serving throughput over existing mechanisms by 1.85X due to maximizing the use of host memory. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08182 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Memory Offloading for Large Language Model Inference with Latency SLO Guarantees Ma, Chenxiang Ye, Zhisheng Zhao, Hanyu Yang, Zehua Fu, Tianhao Han, Jiaxun Zhang, Jie Luo, Yingwei Wang, Xiaolin Wang, Zhenlin Li, Yong Zhou, Diyu Distributed, Parallel, and Cluster Computing Offloading large language models (LLMs) state to host memory during inference promises to reduce operational costs by supporting larger models, longer inputs, and larger batch sizes. However, the design of existing memory offloading mechanisms does not take latency service-level objectives (SLOs) into consideration. As a result, they either lead to frequent SLO violations or underutilize host memory, thereby incurring economic loss and thus defeating the purpose of memory offloading. This paper presents Select-N, a latency-SLO-aware memory offloading system for LLM serving. A key challenge in designing Select-N is to reconcile the tension between meeting SLOs and maximizing host memory usage. Select-N overcomes it by exploiting a unique characteristic of modern LLMs: during serving, the computation time of each decoder layer is deterministic. Leveraging this, Select-N introduces offloading interval, an internal tunable knob that captures the tradeoff between SLOs and host memory usage, thereby reducing the aforementioned challenge to pick an optimal offloading interval. With that, Select-N proposes a two-stage approach to automatically pick the offloading interval. The first stage is offline that generates the range of optimal offloading interval, while the second stage adjusts offloading interval at the granularity of inference iteration based on runtime hardware status. Our evaluation shows that Select-N consistently meets SLOs and improves the serving throughput over existing mechanisms by 1.85X due to maximizing the use of host memory. |
| title | Memory Offloading for Large Language Model Inference with Latency SLO Guarantees |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2502.08182 |