ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912040220622848 |
|---|---|
| author | Lee, Munkyu Seong, Sihoon Kang, Minki Lee, Jihyuk Na, Gap-Joo Chun, In-Geol Nikolopoulos, Dimitrios Hong, Cheol-Ho |
| author_facet | Lee, Munkyu Seong, Sihoon Kang, Minki Lee, Jihyuk Na, Gap-Joo Chun, In-Geol Nikolopoulos, Dimitrios Hong, Cheol-Ho |
| contents | In cloud environments, GPU-based deep neural network (DNN) inference servers are required to meet the Service Level Objective (SLO) latency for each workload under a specified request rate, while also minimizing GPU resource consumption. However, previous studies have not fully achieved this objective. In this paper, we propose ParvaGPU, a technology that facilitates spatial GPU sharing for large-scale DNN inference in cloud computing. ParvaGPU integrates NVIDIA's Multi-Instance GPU (MIG) and Multi-Process Service (MPS) technologies to enhance GPU utilization, with the goal of meeting the diverse SLOs of each workload and reducing overall GPU usage. Specifically, ParvaGPU addresses the challenges of minimizing underutilization within allocated GPU space partitions and external fragmentation in combined MIG and MPS environments. We conducted our assessment on multiple A100 GPUs, evaluating 11 diverse DNN workloads with varying SLOs. Our evaluation revealed no SLO violations and a significant reduction in GPU usage compared to state-of-the-art frameworks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_14447 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments Lee, Munkyu Seong, Sihoon Kang, Minki Lee, Jihyuk Na, Gap-Joo Chun, In-Geol Nikolopoulos, Dimitrios Hong, Cheol-Ho Distributed, Parallel, and Cluster Computing In cloud environments, GPU-based deep neural network (DNN) inference servers are required to meet the Service Level Objective (SLO) latency for each workload under a specified request rate, while also minimizing GPU resource consumption. However, previous studies have not fully achieved this objective. In this paper, we propose ParvaGPU, a technology that facilitates spatial GPU sharing for large-scale DNN inference in cloud computing. ParvaGPU integrates NVIDIA's Multi-Instance GPU (MIG) and Multi-Process Service (MPS) technologies to enhance GPU utilization, with the goal of meeting the diverse SLOs of each workload and reducing overall GPU usage. Specifically, ParvaGPU addresses the challenges of minimizing underutilization within allocated GPU space partitions and external fragmentation in combined MIG and MPS environments. We conducted our assessment on multiple A100 GPUs, evaluating 11 diverse DNN workloads with varying SLOs. Our evaluation revealed no SLO violations and a significant reduction in GPU usage compared to state-of-the-art frameworks. |
| title | ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2409.14447 |