Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.09505 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918124364759040 |
|---|---|
| author | Wang, Zhanghan Ding, Ding Zhu, Hang Lin, Haibin Panda, Aurojit |
| author_facet | Wang, Zhanghan Ding, Ding Zhu, Hang Lin, Haibin Panda, Aurojit |
| contents | Distributed machine learning training and inference is common today because today's large models require more memory and compute than can be provided by a single GPU. Distributed models are generally produced by programmers who take a sequential model specification and apply several distribution strategies to distribute state and computation across GPUs. Unfortunately, bugs can be introduced in the process, and a distributed model implementation's outputs might differ from the sequential model's outputs. In this paper, we describe an approach to statically identify such bugs by checking model refinement, that is, can the sequential model's outputs be reconstructed from the distributed model's outputs? Our approach, implemented in GraphGuard, uses iterative rewriting to prove model refinement. Our approach can scale to today's large models and deployments: we evaluate it using GPT and Llama-3. Further, it provides actionable output that aids in bug localization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09505 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Verify Distributed Deep Learning Model Implementation Refinement with Iterative Relation Inference Wang, Zhanghan Ding, Ding Zhu, Hang Lin, Haibin Panda, Aurojit Distributed, Parallel, and Cluster Computing Artificial Intelligence Distributed machine learning training and inference is common today because today's large models require more memory and compute than can be provided by a single GPU. Distributed models are generally produced by programmers who take a sequential model specification and apply several distribution strategies to distribute state and computation across GPUs. Unfortunately, bugs can be introduced in the process, and a distributed model implementation's outputs might differ from the sequential model's outputs. In this paper, we describe an approach to statically identify such bugs by checking model refinement, that is, can the sequential model's outputs be reconstructed from the distributed model's outputs? Our approach, implemented in GraphGuard, uses iterative rewriting to prove model refinement. Our approach can scale to today's large models and deployments: we evaluate it using GPT and Llama-3. Further, it provides actionable output that aids in bug localization. |
| title | Verify Distributed Deep Learning Model Implementation Refinement with Iterative Relation Inference |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2508.09505 |