Prompt-Aware Scheduling for Low-Latency LLM Serving
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908585487761408 |
|---|---|
| author | Tao, Yiheng Zhang, Yihe Dearing, Matthew T. Wang, Xin Fan, Yuping Lan, Zhiling |
| author_facet | Tao, Yiheng Zhang, Yihe Dearing, Matthew T. Wang, Xin Fan, Yuping Lan, Zhiling |
| contents | Efficient scheduling of LLM inference tasks is essential for achieving low latency and high throughput, particularly with the growing use of reasoning-capable LLMs. Traditional strategies like First-Come-First-Serve (FCFS) often suffer from Head-of-Line (HOL) blocking, where long-running tasks delay shorter ones queued behind them. In this paper, we introduce PARS, a prompt-aware LLM task scheduler that improves serving efficiency by approximating shortest-job-first (SJF) scheduling through pairwise ranking with margin ranking loss. PARS focuses on impactful scheduling decisions and is seamlessly integrated into the state-of-the-art LLM serving system vLLM. It effectively predicts response-length-based task ordering, reducing latency with minimal overhead. Extensive experiments across multiple LLMs and real-world inference datasets show that PARS significantly improves performance, including for reasoning workloads. Furthermore, our cross-model evaluations demonstrate that the design generalizes well, enabling effective scheduling even when predictors are trained on different LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03243 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Prompt-Aware Scheduling for Low-Latency LLM Serving Tao, Yiheng Zhang, Yihe Dearing, Matthew T. Wang, Xin Fan, Yuping Lan, Zhiling Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Performance Efficient scheduling of LLM inference tasks is essential for achieving low latency and high throughput, particularly with the growing use of reasoning-capable LLMs. Traditional strategies like First-Come-First-Serve (FCFS) often suffer from Head-of-Line (HOL) blocking, where long-running tasks delay shorter ones queued behind them. In this paper, we introduce PARS, a prompt-aware LLM task scheduler that improves serving efficiency by approximating shortest-job-first (SJF) scheduling through pairwise ranking with margin ranking loss. PARS focuses on impactful scheduling decisions and is seamlessly integrated into the state-of-the-art LLM serving system vLLM. It effectively predicts response-length-based task ordering, reducing latency with minimal overhead. Extensive experiments across multiple LLMs and real-world inference datasets show that PARS significantly improves performance, including for reasoning workloads. Furthermore, our cross-model evaluations demonstrate that the design generalizes well, enabling effective scheduling even when predictors are trained on different LLMs. |
| title | Prompt-Aware Scheduling for Low-Latency LLM Serving |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Performance |
| url | https://arxiv.org/abs/2510.03243 |