LLM-PRISM: Characterizing Silent Data Corruption from Permanent GPU Faults in LLM Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913024504233984 |
|---|---|
| author | Tyagi, Abhishek Hukerikar, Saurabh Saxena, Nirmal Huang, Yanxiang Shirvani, Philip Tung, Chung-Hsuan Zhu, Yuhao |
| author_facet | Tyagi, Abhishek Hukerikar, Saurabh Saxena, Nirmal Huang, Yanxiang Shirvani, Philip Tung, Chung-Hsuan Zhu, Yuhao |
| contents | Large-scale LLM training is increasingly susceptible to hardware defects stemming from manufacturing escapes and silicon aging. These defects manifest as Silent Data Corruption (SDC) that perturb gradients and parameters throughout the training process. We present LLM-PRISM, a methodology to characterize LLM pre-training resilience to hardware faults. LLM-PRISM couples RTL-level GPU fault simulation with a stochastic injection engine embedded in Megatron-LM. Through 7,664 training runs across FP16, BF16, and FP8 regimes, we analyze how fault type, rate, and numeric format govern resilience. We find that while LLMs resist low-frequency faults, impact is highly non-uniform; critical datapaths and specific precision formats can induce catastrophic divergence even at moderate fault rates. This study provides the first hardware-grounded, pre-training characterization of SDC resilience. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_10390 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LLM-PRISM: Characterizing Silent Data Corruption from Permanent GPU Faults in LLM Training Tyagi, Abhishek Hukerikar, Saurabh Saxena, Nirmal Huang, Yanxiang Shirvani, Philip Tung, Chung-Hsuan Zhu, Yuhao Hardware Architecture Large-scale LLM training is increasingly susceptible to hardware defects stemming from manufacturing escapes and silicon aging. These defects manifest as Silent Data Corruption (SDC) that perturb gradients and parameters throughout the training process. We present LLM-PRISM, a methodology to characterize LLM pre-training resilience to hardware faults. LLM-PRISM couples RTL-level GPU fault simulation with a stochastic injection engine embedded in Megatron-LM. Through 7,664 training runs across FP16, BF16, and FP8 regimes, we analyze how fault type, rate, and numeric format govern resilience. We find that while LLMs resist low-frequency faults, impact is highly non-uniform; critical datapaths and specific precision formats can induce catastrophic divergence even at moderate fault rates. This study provides the first hardware-grounded, pre-training characterization of SDC resilience. |
| title | LLM-PRISM: Characterizing Silent Data Corruption from Permanent GPU Faults in LLM Training |
| topic | Hardware Architecture |
| url | https://arxiv.org/abs/2604.10390 |