Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911513823936512 |
|---|---|
| author | Yang, Mingyan Wang, Guanjie Luo, Manqi Liu, Yifei Chen, Chen Zhao, Han Feng, Yu Chen, Quan Guo, Minyi |
| author_facet | Yang, Mingyan Wang, Guanjie Luo, Manqi Liu, Yifei Chen, Chen Zhao, Han Feng, Yu Chen, Quan Guo, Minyi |
| contents | LLM agents, which often comprise parallel inference tasks, are commonly adopted to solve real-world problems. When serving such task-parallel LLM agents in shared GPU servers, the scheduler is expected to attain fast agent completion with guaranteed worst-case performance. For that objective, our insight is to selectively pampering agents based on their completion order under idealized fair-sharing. We design Justitia, a fair and also efficient scheduler for task-parallel LLM agents. Noticing that memory is prevalently a bottleneck in LLM serving, Justitia quantifies the true agent cost in a memory-centric manner. It also adopts a light-weight yet accurate method to predict agent costs. Finally, Justitia adopts a virtual-time based fair queuing algorithm to reduce the overall performance with guaranteed worst-case delay. We have implemented Justitia atop vLLM, and experimental results involving diverse agents show that it can substantially enhance the scheduling efficiency with fairness preserved. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_17015 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering Yang, Mingyan Wang, Guanjie Luo, Manqi Liu, Yifei Chen, Chen Zhao, Han Feng, Yu Chen, Quan Guo, Minyi Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing LLM agents, which often comprise parallel inference tasks, are commonly adopted to solve real-world problems. When serving such task-parallel LLM agents in shared GPU servers, the scheduler is expected to attain fast agent completion with guaranteed worst-case performance. For that objective, our insight is to selectively pampering agents based on their completion order under idealized fair-sharing. We design Justitia, a fair and also efficient scheduler for task-parallel LLM agents. Noticing that memory is prevalently a bottleneck in LLM serving, Justitia quantifies the true agent cost in a memory-centric manner. It also adopts a light-weight yet accurate method to predict agent costs. Finally, Justitia adopts a virtual-time based fair queuing algorithm to reduce the overall performance with guaranteed worst-case delay. We have implemented Justitia atop vLLM, and experimental results involving diverse agents show that it can substantially enhance the scheduling efficiency with fairness preserved. |
| title | Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2510.17015 |