Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911762224250880 |
|---|---|
| author | Li, Bingbing Yuan, Geng Wang, Zigeng Huang, Shaoyi Peng, Hongwu Behnam, Payman Wen, Wujie Liu, Hang Ding, Caiwen |
| author_facet | Li, Bingbing Yuan, Geng Wang, Zigeng Huang, Shaoyi Peng, Hongwu Behnam, Payman Wen, Wujie Liu, Hang Ding, Caiwen |
| contents | Resistive Random Access Memory (ReRAM) has emerged as a promising platform for deep neural networks (DNNs) due to its support for parallel in-situ matrix-vector multiplication. However, hardware failures, such as stuck-at-fault defects, can result in significant prediction errors during model inference. While additional crossbars can be used to address these failures, they come with storage overhead and are not efficient in terms of space, energy, and cost. In this paper, we propose a fault protection mechanism that incurs zero space cost. Our approach includes: 1) differentiable structure pruning of rows and columns to reduce model redundancy, 2) weight duplication and voting for robust output, and 3) embedding duplicated most significant bits (MSBs) into the model weight. We evaluate our method on nine tasks of the GLUE benchmark with the BERT model, and experimental results prove its effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_11664 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM Li, Bingbing Yuan, Geng Wang, Zigeng Huang, Shaoyi Peng, Hongwu Behnam, Payman Wen, Wujie Liu, Hang Ding, Caiwen Machine Learning Artificial Intelligence Hardware Architecture Resistive Random Access Memory (ReRAM) has emerged as a promising platform for deep neural networks (DNNs) due to its support for parallel in-situ matrix-vector multiplication. However, hardware failures, such as stuck-at-fault defects, can result in significant prediction errors during model inference. While additional crossbars can be used to address these failures, they come with storage overhead and are not efficient in terms of space, energy, and cost. In this paper, we propose a fault protection mechanism that incurs zero space cost. Our approach includes: 1) differentiable structure pruning of rows and columns to reduce model redundancy, 2) weight duplication and voting for robust output, and 3) embedding duplicated most significant bits (MSBs) into the model weight. We evaluate our method on nine tasks of the GLUE benchmark with the BERT model, and experimental results prove its effectiveness. |
| title | Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM |
| topic | Machine Learning Artificial Intelligence Hardware Architecture |
| url | https://arxiv.org/abs/2401.11664 |