Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913899303927808 |
|---|---|
| author | Jiang, Yuxuan Zhou, Ziming Xu, Boyu Liu, Beijie Xu, Runhui Huang, Peng |
| author_facet | Jiang, Yuxuan Zhou, Ziming Xu, Boyu Liu, Beijie Xu, Runhui Huang, Peng |
| contents | Training deep learning (DL) models is a complex process, making it prone to silent errors that are challenging to detect and diagnose. This paper presents TRAINCHECK, a framework that takes a proactive checking approach to address silent training errors. TRAINCHECK automatically infers invariants tailored for DL training. It uses these invariants to proactively detect silent errors during the training process while providing debugging help. To evaluate TRAINCHECK, we reproduce 20 real-world silent training errors with diverse root causes. TRAINCHECK successfully detects 18 errors within a single training iteration. It also uncovers 6 unknown bugs in popular training libraries that lead to silent errors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14813 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks Jiang, Yuxuan Zhou, Ziming Xu, Boyu Liu, Beijie Xu, Runhui Huang, Peng Machine Learning Artificial Intelligence Training deep learning (DL) models is a complex process, making it prone to silent errors that are challenging to detect and diagnose. This paper presents TRAINCHECK, a framework that takes a proactive checking approach to address silent training errors. TRAINCHECK automatically infers invariants tailored for DL training. It uses these invariants to proactively detect silent errors during the training process while providing debugging help. To evaluate TRAINCHECK, we reproduce 20 real-world silent training errors with diverse root causes. TRAINCHECK successfully detects 18 errors within a single training iteration. It also uncovers 6 unknown bugs in popular training libraries that lead to silent errors. |
| title | Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.14813 |