Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.15919 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913058092220416 |
|---|---|
| author | Yousefijamarani, Zahra Wang, Xinglu Wang, Qian Heisler, Morgan Lindsay Shabani, Taha Gholipour, Niloofar Yassini, Parham Chang, Hong Chen, Kan Zhang, Qiantao Bai, Xiaolong Wang, Jiannan Xiong, Ying Zhang, Yong Fan, Zhenan |
| author_facet | Yousefijamarani, Zahra Wang, Xinglu Wang, Qian Heisler, Morgan Lindsay Shabani, Taha Gholipour, Niloofar Yassini, Parham Chang, Hong Chen, Kan Zhang, Qiantao Bai, Xiaolong Wang, Jiannan Xiong, Ying Zhang, Yong Fan, Zhenan |
| contents | Large language model (LLM) serving faces the dual challenge of meeting strict user-specific service-level objectives (SLOs) while minimizing computational cost under dynamic, multi-task workloads. Existing approaches either rely on static scheduling policies or focus on single-task settings, limiting their applicability in real-world deployments with heterogeneous requests, variable prompt lengths, and elastic scaling requirements.
We present HFX, a production LLM serving system that jointly optimizes request scheduling and elastic scaling across model replicas to satisfy diverse SLOs. HFX introduces a \textbf{scheduler} that performs proactive budget estimation and prioritization to ensure SLO compliance for both new and in-flight requests. HFX also integrates a \textbf{scaler} that supports fast device-to-device (D2D) weight transfer, reducing cold-start latency. Additionally, the system supports both colocated and disaggregated prefill/decode deployments, enabling adaptation to diverse workload patterns and cloud environments.
Through extensive experiments on multi-task workloads, we demonstrate consistently higher SLO attainment, lower end-to-end latency, and lower NPU usage cost by up to 4.44$\times$, 65.82\%, and 49.81\%, respectively, compared to state-of-the-art systems. Our results highlight the effectiveness of SLO-aware scheduling and scaling in practical LLM serving, providing a robust framework for cost-efficient and SLO-compliant deployments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_15919 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling Yousefijamarani, Zahra Wang, Xinglu Wang, Qian Heisler, Morgan Lindsay Shabani, Taha Gholipour, Niloofar Yassini, Parham Chang, Hong Chen, Kan Zhang, Qiantao Bai, Xiaolong Wang, Jiannan Xiong, Ying Zhang, Yong Fan, Zhenan Distributed, Parallel, and Cluster Computing Artificial Intelligence Large language model (LLM) serving faces the dual challenge of meeting strict user-specific service-level objectives (SLOs) while minimizing computational cost under dynamic, multi-task workloads. Existing approaches either rely on static scheduling policies or focus on single-task settings, limiting their applicability in real-world deployments with heterogeneous requests, variable prompt lengths, and elastic scaling requirements. We present HFX, a production LLM serving system that jointly optimizes request scheduling and elastic scaling across model replicas to satisfy diverse SLOs. HFX introduces a \textbf{scheduler} that performs proactive budget estimation and prioritization to ensure SLO compliance for both new and in-flight requests. HFX also integrates a \textbf{scaler} that supports fast device-to-device (D2D) weight transfer, reducing cold-start latency. Additionally, the system supports both colocated and disaggregated prefill/decode deployments, enabling adaptation to diverse workload patterns and cloud environments. Through extensive experiments on multi-task workloads, we demonstrate consistently higher SLO attainment, lower end-to-end latency, and lower NPU usage cost by up to 4.44$\times$, 65.82\%, and 49.81\%, respectively, compared to state-of-the-art systems. Our results highlight the effectiveness of SLO-aware scheduling and scaling in practical LLM serving, providing a robust framework for cost-efficient and SLO-compliant deployments. |
| title | HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2508.15919 |