Equinox: Holistic Fair Scheduling in Serving Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912549516083200 |
|---|---|
| author | Wei, Zhixiang Yen, James Chen, Jingyi Zhang, Ziyang Huang, Zhibai Chen, Chen Yu, Xingzi Gu, Yicheng Wu, Chenggang Wang, Yun Xia, Mingyuan Wu, Jie Wang, Hao Qi, Zhengwei |
| author_facet | Wei, Zhixiang Yen, James Chen, Jingyi Zhang, Ziyang Huang, Zhibai Chen, Chen Yu, Xingzi Gu, Yicheng Wu, Chenggang Wang, Yun Xia, Mingyuan Wu, Jie Wang, Hao Qi, Zhengwei |
| contents | We address the limitations of current LLM serving with a dual-counter framework separating user and operator perspectives. The User Fairness Counter measures quality of service via weighted tokens and latency; the Resource Fairness Counter measures operational efficiency through throughput and GPU utilization. Since these metrics are only available post-execution, creating a scheduling paradox, we introduce a deterministic Mixture of Prediction Experts (MoPE) framework to predict user-perceived latency, output tokens, throughput, and GPU utilization. These predictions enable calculation of a unified Holistic Fairness score that balances both counters through tunable parameters for proactive fairness-aware scheduling. We implement this in Equinox, an open-source system with other optimizations like adaptive batching, and stall-free scheduling. Evaluations on production traces (ShareGPT, LMSYS) and synthetic workloads demonstrate Equinox achieves up to $1.3\times$ higher throughput, 60\% lower time-to-first-token latency, and 13\% higher fairness versus VTC while maintaining 94\% GPU utilization, proving fairness under bounded discrepancy across heterogeneous platforms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_16646 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Equinox: Holistic Fair Scheduling in Serving Large Language Models Wei, Zhixiang Yen, James Chen, Jingyi Zhang, Ziyang Huang, Zhibai Chen, Chen Yu, Xingzi Gu, Yicheng Wu, Chenggang Wang, Yun Xia, Mingyuan Wu, Jie Wang, Hao Qi, Zhengwei Distributed, Parallel, and Cluster Computing Artificial Intelligence We address the limitations of current LLM serving with a dual-counter framework separating user and operator perspectives. The User Fairness Counter measures quality of service via weighted tokens and latency; the Resource Fairness Counter measures operational efficiency through throughput and GPU utilization. Since these metrics are only available post-execution, creating a scheduling paradox, we introduce a deterministic Mixture of Prediction Experts (MoPE) framework to predict user-perceived latency, output tokens, throughput, and GPU utilization. These predictions enable calculation of a unified Holistic Fairness score that balances both counters through tunable parameters for proactive fairness-aware scheduling. We implement this in Equinox, an open-source system with other optimizations like adaptive batching, and stall-free scheduling. Evaluations on production traces (ShareGPT, LMSYS) and synthetic workloads demonstrate Equinox achieves up to $1.3\times$ higher throughput, 60\% lower time-to-first-token latency, and 13\% higher fairness versus VTC while maintaining 94\% GPU utilization, proving fairness under bounded discrepancy across heterogeneous platforms. |
| title | Equinox: Holistic Fair Scheduling in Serving Large Language Models |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2508.16646 |