Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908631933386752 |
|---|---|
| author | Wu, Yu Liu, Tongxuan Zeng, Yuting Wu, Siyu Xiong, Jun Dong, Xianzhe Yang, Hailong Zhang, Ke Li, Jing |
| author_facet | Wu, Yu Liu, Tongxuan Zeng, Yuting Wu, Siyu Xiong, Jun Dong, Xianzhe Yang, Hailong Zhang, Ke Li, Jing |
| contents | Existing large language model (LLM) serving systems typically employ Prefill-Decode disaggregated architecture to prevent computational interference between the prefill and decode phases. However, in real-world LLM serving scenarios, significant fluctuations in request input/output lengths lead to imbalanced computational loads between prefill and decode nodes under traditional static node allocation strategies, consequently preventing efficient utilization of computing resources to improve the system's goodput. To address this challenge, we design and implement Arrow, an adaptive scheduler that leverages stateless instances and latency characteristics of prefill and decode tasks to achieve efficient adaptive request and instance scheduling. Arrow dynamically adjusts the number of instances handling prefill and decode tasks based on real-time cluster performance metrics, substantially enhancing the system's capability to handle traffic spikes and load variations. Our evaluation under diverse real-world workloads shows that Arrow achieves up to $2.55 \times$ higher request serving rates compared to state-of-the-art Prefill-Decode disaggregated serving systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11916 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture Wu, Yu Liu, Tongxuan Zeng, Yuting Wu, Siyu Xiong, Jun Dong, Xianzhe Yang, Hailong Zhang, Ke Li, Jing Distributed, Parallel, and Cluster Computing Existing large language model (LLM) serving systems typically employ Prefill-Decode disaggregated architecture to prevent computational interference between the prefill and decode phases. However, in real-world LLM serving scenarios, significant fluctuations in request input/output lengths lead to imbalanced computational loads between prefill and decode nodes under traditional static node allocation strategies, consequently preventing efficient utilization of computing resources to improve the system's goodput. To address this challenge, we design and implement Arrow, an adaptive scheduler that leverages stateless instances and latency characteristics of prefill and decode tasks to achieve efficient adaptive request and instance scheduling. Arrow dynamically adjusts the number of instances handling prefill and decode tasks based on real-time cluster performance metrics, substantially enhancing the system's capability to handle traffic spikes and load variations. Our evaluation under diverse real-world workloads shows that Arrow achieves up to $2.55 \times$ higher request serving rates compared to state-of-the-art Prefill-Decode disaggregated serving systems. |
| title | Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2505.11916 |