Characterization-Guided GPU Fault Resilience in NVIDIA MPS
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911717993218048 |
|---|---|
| author | Liu, Rixin Cui, Xingqi Wang, Kaijian Ding, Xinheng Liu, Zirui Wang, Yuke Xing, Jiarong |
| author_facet | Liu, Rixin Cui, Xingqi Wang, Kaijian Ding, Xinheng Liu, Zirui Wang, Yuke Xing, Jiarong |
| contents | NVIDIA Multi-Process Service (MPS) enables fine-grained GPU sharing by allowing multiple processes to execute concurrently on the same GPU, making it an important mechanism for improving GPU utilization. However, MPS has weak fault resilience: a fault in one process can terminate all co-running processes, limiting its adoption in resilience-critical settings such as multi-tenant GPU clusters. In this work, we design fault-resilient MPS to solve this problem. Our design is guided by insights from a systematic characterization of GPU faults and a deep analysis of their end-to-end processing pipeline. Based on these insights, we design two complementary mechanisms. A fault isolation mechanism for the dominant memory-related faults that can be fully isolated by software intervention in the open GPU driver kernel module. For other faults whose process is within proprietary software, we design a practical mechanism -- fast recovery using virtual memory based GPU-resident state sharing. Our evaluation on different GPUs and workloads shows that these mechanisms can handle corresponding faults effectively with minimal overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_26461 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Characterization-Guided GPU Fault Resilience in NVIDIA MPS Liu, Rixin Cui, Xingqi Wang, Kaijian Ding, Xinheng Liu, Zirui Wang, Yuke Xing, Jiarong Distributed, Parallel, and Cluster Computing NVIDIA Multi-Process Service (MPS) enables fine-grained GPU sharing by allowing multiple processes to execute concurrently on the same GPU, making it an important mechanism for improving GPU utilization. However, MPS has weak fault resilience: a fault in one process can terminate all co-running processes, limiting its adoption in resilience-critical settings such as multi-tenant GPU clusters. In this work, we design fault-resilient MPS to solve this problem. Our design is guided by insights from a systematic characterization of GPU faults and a deep analysis of their end-to-end processing pipeline. Based on these insights, we design two complementary mechanisms. A fault isolation mechanism for the dominant memory-related faults that can be fully isolated by software intervention in the open GPU driver kernel module. For other faults whose process is within proprietary software, we design a practical mechanism -- fast recovery using virtual memory based GPU-resident state sharing. Our evaluation on different GPUs and workloads shows that these mechanisms can handle corresponding faults effectively with minimal overhead. |
| title | Characterization-Guided GPU Fault Resilience in NVIDIA MPS |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2605.26461 |