SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911511149019136 |
|---|---|
| author | Gan, Zhenghao Bao, Yichen Liu, Yifei Chen, Chen Chen, Quan Guo, Minyi |
| author_facet | Gan, Zhenghao Bao, Yichen Liu, Yifei Chen, Chen Chen, Quan Guo, Minyi |
| contents | Efficient LLM inference scheduling is crucial for user experience. However, LLM inferences exhibit remarkable demand uncertainty (with unknown output length beforehand) and hybridity (being both compute and memory intensive). Existing LLM schedulers rely on simple heuristics or focus purely on compute resource, suffering suboptimal performance.
In this work, we propose SageSched, an efficient LLM scheduler that properly handles demand uncertainty and hybridity of inference workloads. SageSched combines prompt contents with the past inference results to predict output-length distribution in a light-weight and also accurate manner. Meanwhile, it models the true service cost of an inference request with both compute and memory aspects considered. Finally, SageSched employs an uncertainty-aware scheduling policy that can yield the best overall efficiency given the request cost distributions. Testbed experiments over diverse setups confirm that SageSched can attain an efficiency improvement of over 28.7%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07917 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity Gan, Zhenghao Bao, Yichen Liu, Yifei Chen, Chen Chen, Quan Guo, Minyi Distributed, Parallel, and Cluster Computing Efficient LLM inference scheduling is crucial for user experience. However, LLM inferences exhibit remarkable demand uncertainty (with unknown output length beforehand) and hybridity (being both compute and memory intensive). Existing LLM schedulers rely on simple heuristics or focus purely on compute resource, suffering suboptimal performance. In this work, we propose SageSched, an efficient LLM scheduler that properly handles demand uncertainty and hybridity of inference workloads. SageSched combines prompt contents with the past inference results to predict output-length distribution in a light-weight and also accurate manner. Meanwhile, it models the true service cost of an inference request with both compute and memory aspects considered. Finally, SageSched employs an uncertainty-aware scheduling policy that can yield the best overall efficiency given the request cost distributions. Testbed experiments over diverse setups confirm that SageSched can attain an efficiency improvement of over 28.7%. |
| title | SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2603.07917 |