Valve: Production Online-Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911578258931712 |
|---|---|
| author | Liu, Fangyue Liu, Hua Lyu, Xinyuan Ai, Shuo Liang, Hao Chen, Lingpeng Hu, Ziqian Zha, Chong Jin, Xin Luo, Hanmei Chen, Peng |
| author_facet | Liu, Fangyue Liu, Hua Lyu, Xinyuan Ai, Shuo Liang, Hao Chen, Lingpeng Hu, Ziqian Zha, Chong Jin, Xin Luo, Hanmei Chen, Peng |
| contents | LLM inference powers latency-critical production services nowadays. The bursty nature of inference traffic results in over-provisioning, which in turn leads to resource underutilization. While online-offline colocation promises to utilize idle capacity, broad production deployment must overcome two major challenges: (i) large online interference due to slow or frequent preemptions, and (ii) extensive frameworks and drivers modifications, to colocate different models and support preemptions. We present Valve, a production-friendly colocation system that jointly bounds preemption latency and preemption rate. Specifically, Valve enables sub-millisecond compute preemption at most once per online request, and rate-limited sub-layer memory reclamation. These guaranties are provided by a GPU runtime that combines channel-controlled compute isolation, page-fault-free memory reclamation, and dynamic memory reservation. Critically, Valve is practical to deploy, requiring one line of driver modification and 20 lines of framework patch. Deployed on 8,054 GPUs in production, Valve improves cluster utilization by 34.6%, which translates to a 2,170 GPU save. This efficiency gains is achieved with minimal online interference, incurring <5% TTFT increase and <2% TPOT increase across workloads. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07874 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Valve: Production Online-Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate Liu, Fangyue Liu, Hua Lyu, Xinyuan Ai, Shuo Liang, Hao Chen, Lingpeng Hu, Ziqian Zha, Chong Jin, Xin Luo, Hanmei Chen, Peng Operating Systems LLM inference powers latency-critical production services nowadays. The bursty nature of inference traffic results in over-provisioning, which in turn leads to resource underutilization. While online-offline colocation promises to utilize idle capacity, broad production deployment must overcome two major challenges: (i) large online interference due to slow or frequent preemptions, and (ii) extensive frameworks and drivers modifications, to colocate different models and support preemptions. We present Valve, a production-friendly colocation system that jointly bounds preemption latency and preemption rate. Specifically, Valve enables sub-millisecond compute preemption at most once per online request, and rate-limited sub-layer memory reclamation. These guaranties are provided by a GPU runtime that combines channel-controlled compute isolation, page-fault-free memory reclamation, and dynamic memory reservation. Critically, Valve is practical to deploy, requiring one line of driver modification and 20 lines of framework patch. Deployed on 8,054 GPUs in production, Valve improves cluster utilization by 34.6%, which translates to a 2,170 GPU save. This efficiency gains is achieved with minimal online interference, incurring <5% TTFT increase and <2% TPOT increase across workloads. |
| title | Valve: Production Online-Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate |
| topic | Operating Systems |
| url | https://arxiv.org/abs/2604.07874 |