Adaptive Request Scheduling for CodeLLM Serving with SLA Guarantees
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912448860127232 |
|---|---|
| author | Chang, Shi Chen, Boyuan Thangarajah, Kishanthan Lutfiyya, Hanan Hassan, Ahmed E. |
| author_facet | Chang, Shi Chen, Boyuan Thangarajah, Kishanthan Lutfiyya, Hanan Hassan, Ahmed E. |
| contents | Code Large Language Models (CodeLLMs) are increasingly integrated into modern software development workflows, yet efficiently serving them in resource-constrained, self-hosted environments remains a significant challenge. Existing LLM serving systems employs Continuous Batching for throughput improvement. However, they rely on static batch size configurations that cannot adapt to fluctuating request rates or heterogeneous workloads, leading to frequent SLA (Service Level Agreement) violations and unstable performance. In this study, We propose SABER, a dynamic batching strategy that predicts per-request SLA feasibility and adjusts decisions in real time. SABER improves goodput by up to 26% over the best static configurations and reduces latency variability by up to 45%, all without manual tuning or service restarts. Our results demonstrate that SLA-aware, adaptive scheduling is key to robust, high-performance CodeLLM serving. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_19677 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adaptive Request Scheduling for CodeLLM Serving with SLA Guarantees Chang, Shi Chen, Boyuan Thangarajah, Kishanthan Lutfiyya, Hanan Hassan, Ahmed E. Software Engineering Code Large Language Models (CodeLLMs) are increasingly integrated into modern software development workflows, yet efficiently serving them in resource-constrained, self-hosted environments remains a significant challenge. Existing LLM serving systems employs Continuous Batching for throughput improvement. However, they rely on static batch size configurations that cannot adapt to fluctuating request rates or heterogeneous workloads, leading to frequent SLA (Service Level Agreement) violations and unstable performance. In this study, We propose SABER, a dynamic batching strategy that predicts per-request SLA feasibility and adjusts decisions in real time. SABER improves goodput by up to 26% over the best static configurations and reduces latency variability by up to 45%, all without manual tuning or service restarts. Our results demonstrate that SLA-aware, adaptive scheduling is key to robust, high-performance CodeLLM serving. |
| title | Adaptive Request Scheduling for CodeLLM Serving with SLA Guarantees |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2506.19677 |