PARD: Enhancing Goodput for Inference Pipeline via Proactive Request Dropping
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915786439786496 |
|---|---|
| author | Zhao, Zhixin Hu, Yitao Chen, Simin Ji, Mingfang Yang, Wei Zhang, Yuhao Zhao, Laiping Li, Wenxin Liu, Xiulong Qu, Wenyu Wang, Hao |
| author_facet | Zhao, Zhixin Hu, Yitao Chen, Simin Ji, Mingfang Yang, Wei Zhang, Yuhao Zhao, Laiping Li, Wenxin Liu, Xiulong Qu, Wenyu Wang, Hao |
| contents | Modern deep neural network (DNN) applications integrate multiple DNN models into inference pipelines with stringent latency requirements for customized tasks. To mitigate extensive request timeouts caused by accumulation, systems for inference pipelines commonly drop a subset of requests so the remaining ones can satisfy latency constraints. Since it is commonly believed that request dropping adversely affects goodput, existing systems only drop requests when they have to, which we call reactive dropping. However, this reactive policy can not maintain high goodput, as it neither makes timely dropping decisions nor identifies the proper set of requests to drop, leading to issues of dropping requests too late or dropping the wrong set of requests.
We propose that the inference system should proactively drop certain requests in advance to enhance the goodput across the entire workload. To achieve this, we design an inference system PARD. It enhances goodput with timely and precise dropping decisions by integrating a proactive dropping method that decides when to drop requests using runtime information of the inference pipeline, and an adaptive request priority mechanism that selects which specific requests to drop based on remaining latency budgets and workload intensity. Evaluation on a cluster of 64 GPUs over real-world workloads shows that PARD achieves $16\%$-$176\%$ higher goodput than the state of the art while reducing the drop rate and wasted computation resources by $1.6\times$-$17\times$ and $1.5\times$-$62\times$ respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_08747 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PARD: Enhancing Goodput for Inference Pipeline via Proactive Request Dropping Zhao, Zhixin Hu, Yitao Chen, Simin Ji, Mingfang Yang, Wei Zhang, Yuhao Zhao, Laiping Li, Wenxin Liu, Xiulong Qu, Wenyu Wang, Hao Distributed, Parallel, and Cluster Computing Modern deep neural network (DNN) applications integrate multiple DNN models into inference pipelines with stringent latency requirements for customized tasks. To mitigate extensive request timeouts caused by accumulation, systems for inference pipelines commonly drop a subset of requests so the remaining ones can satisfy latency constraints. Since it is commonly believed that request dropping adversely affects goodput, existing systems only drop requests when they have to, which we call reactive dropping. However, this reactive policy can not maintain high goodput, as it neither makes timely dropping decisions nor identifies the proper set of requests to drop, leading to issues of dropping requests too late or dropping the wrong set of requests. We propose that the inference system should proactively drop certain requests in advance to enhance the goodput across the entire workload. To achieve this, we design an inference system PARD. It enhances goodput with timely and precise dropping decisions by integrating a proactive dropping method that decides when to drop requests using runtime information of the inference pipeline, and an adaptive request priority mechanism that selects which specific requests to drop based on remaining latency budgets and workload intensity. Evaluation on a cluster of 64 GPUs over real-world workloads shows that PARD achieves $16\%$-$176\%$ higher goodput than the state of the art while reducing the drop rate and wasted computation resources by $1.6\times$-$17\times$ and $1.5\times$-$62\times$ respectively. |
| title | PARD: Enhancing Goodput for Inference Pipeline via Proactive Request Dropping |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2602.08747 |