EdgeServing: Deadline-Aware Multi-DNN Serving at the Edge
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915986831048704 |
|---|---|
| author | Cao, Jiahe Li, Xiaomeng Liu, Qiang Han, Tao Zhang, Ning Shi, Weisong |
| author_facet | Cao, Jiahe Li, Xiaomeng Liu, Qiang Han, Tao Zhang, Ning Shi, Weisong |
| contents | As edge computing expands, serving multiple deep neural network (DNN) models on a single shared GPU has become a common yet challenging scenario, where each scheduling decision affects the tail latency of all concurrent queues. Existing schedulers rely on local heuristics and fail to capture this global impact, while GPU spatial-sharing approaches sacrifice latency predictability. In this paper, we propose EdgeServing, a deadline-aware multi-DNN serving system for edge devices. EdgeServing adopts time-division GPU sharing with early-exit inference for high inference predictability, and introduces a stability score to quantify how each candidate scheduling decision impacts the future queue status. At runtime, it cohesively selects the model, exit point, and batch size to minimize predicted system-wide SLO impact. Experimental results on multiple hardware platforms show that EdgeServing consistently outperforms representative baselines in both SLO violation ratio and P95 latency, enabled by early-exit mechanism, which expands the scheduling action space under tight latency constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_05527 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | EdgeServing: Deadline-Aware Multi-DNN Serving at the Edge Cao, Jiahe Li, Xiaomeng Liu, Qiang Han, Tao Zhang, Ning Shi, Weisong Distributed, Parallel, and Cluster Computing As edge computing expands, serving multiple deep neural network (DNN) models on a single shared GPU has become a common yet challenging scenario, where each scheduling decision affects the tail latency of all concurrent queues. Existing schedulers rely on local heuristics and fail to capture this global impact, while GPU spatial-sharing approaches sacrifice latency predictability. In this paper, we propose EdgeServing, a deadline-aware multi-DNN serving system for edge devices. EdgeServing adopts time-division GPU sharing with early-exit inference for high inference predictability, and introduces a stability score to quantify how each candidate scheduling decision impacts the future queue status. At runtime, it cohesively selects the model, exit point, and batch size to minimize predicted system-wide SLO impact. Experimental results on multiple hardware platforms show that EdgeServing consistently outperforms representative baselines in both SLO violation ratio and P95 latency, enabled by early-exit mechanism, which expands the scheduling action space under tight latency constraints. |
| title | EdgeServing: Deadline-Aware Multi-DNN Serving at the Edge |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2605.05527 |