WVA: A Global Optimization Control Plane for llmd
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917353885794304 |
|---|---|
| author | Malvankar, Abhishek Villard, Lionel Abdi, Mohammed Sgreccia, Tommaso Shindin, Evgeny Dumba, Braulio Ramani, Vishakha Tantawi, Asser Eilam, Tamar |
| author_facet | Malvankar, Abhishek Villard, Lionel Abdi, Mohammed Sgreccia, Tommaso Shindin, Evgeny Dumba, Braulio Ramani, Vishakha Tantawi, Asser Eilam, Tamar |
| contents | As Large Language Models (LLMs) scale to handle massive concurrent traffic, optimizing the infrastructure required for inference has become a primary challenge. To manage the high cost of GPU resources while ensuring strict service-level objectives (SLOs), operators increasingly deploy models across heterogeneous hardware clusters that multiplex latency-sensitive online requests and throughput-oriented offline requests. However, traditional resource-centric autoscalers like the Kubernetes horizontal pod autoscaler (HPA) do not consider application-specific SLOs, hardware heterogeneity, or internal engine state (like KV cache utilization) globally. This leads to unnecessary scaling, severe resource underutilization, and disrupted stateful inference. To address these limitations, we introduce the Workload Variant Autoscaler (WVA), a specialized control plane co-designed with \texttt{llmd} that tightly couples scaling decisions with the inference server's internal saturation state. By utilizing proactive headroom-based scaling and fragmentation-aware scale-down, our experiments demonstrate that WVA achieves a \textbf{37\% improvement in effective throughput} and a \textbf{10x reduction in request failures} compared to HPA. Furthermore, WVA's cost-aware tiering intrinsically reduces overall power consumption by prioritizing lower-cost, energy-efficient hardware variants over homogeneous scaling on high-end accelerators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_09730 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | WVA: A Global Optimization Control Plane for llmd Malvankar, Abhishek Villard, Lionel Abdi, Mohammed Sgreccia, Tommaso Shindin, Evgeny Dumba, Braulio Ramani, Vishakha Tantawi, Asser Eilam, Tamar Emerging Technologies As Large Language Models (LLMs) scale to handle massive concurrent traffic, optimizing the infrastructure required for inference has become a primary challenge. To manage the high cost of GPU resources while ensuring strict service-level objectives (SLOs), operators increasingly deploy models across heterogeneous hardware clusters that multiplex latency-sensitive online requests and throughput-oriented offline requests. However, traditional resource-centric autoscalers like the Kubernetes horizontal pod autoscaler (HPA) do not consider application-specific SLOs, hardware heterogeneity, or internal engine state (like KV cache utilization) globally. This leads to unnecessary scaling, severe resource underutilization, and disrupted stateful inference. To address these limitations, we introduce the Workload Variant Autoscaler (WVA), a specialized control plane co-designed with \texttt{llmd} that tightly couples scaling decisions with the inference server's internal saturation state. By utilizing proactive headroom-based scaling and fragmentation-aware scale-down, our experiments demonstrate that WVA achieves a \textbf{37\% improvement in effective throughput} and a \textbf{10x reduction in request failures} compared to HPA. Furthermore, WVA's cost-aware tiering intrinsically reduces overall power consumption by prioritizing lower-cost, energy-efficient hardware variants over homogeneous scaling on high-end accelerators. |
| title | WVA: A Global Optimization Control Plane for llmd |
| topic | Emerging Technologies |
| url | https://arxiv.org/abs/2603.09730 |