Bulk-Switching Memristor-based Compute-In-Memory Module for Deep Neural Network Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917590471802880 |
|---|---|
| author | Wu, Yuting Wang, Qiwen Wang, Ziyu Wang, Xinxin Ayyagari, Buvna Krishnan, Siddarth Chudzik, Michael Lu, Wei D. |
| author_facet | Wu, Yuting Wang, Qiwen Wang, Ziyu Wang, Xinxin Ayyagari, Buvna Krishnan, Siddarth Chudzik, Michael Lu, Wei D. |
| contents | The need for deep neural network (DNN) models with higher performance and better functionality leads to the proliferation of very large models. Model training, however, requires intensive computation time and energy. Memristor-based compute-in-memory (CIM) modules can perform vector-matrix multiplication (VMM) in situ and in parallel, and have shown great promises in DNN inference applications. However, CIM-based model training faces challenges due to non-linear weight updates, device variations, and low-precision in analog computing circuits. In this work, we experimentally implement a mixed-precision training scheme to mitigate these effects using a bulk-switching memristor CIM module. Lowprecision CIM modules are used to accelerate the expensive VMM operations, with high precision weight updates accumulated in digital units. Memristor devices are only changed when the accumulated weight update value exceeds a pre-defined threshold. The proposed scheme is implemented with a system-on-chip (SoC) of fully integrated analog CIM modules and digital sub-systems, showing fast convergence of LeNet training to 97.73%. The efficacy of training larger models is evaluated using realistic hardware parameters and shows that that analog CIM modules can enable efficient mix-precision DNN training with accuracy comparable to full-precision software trained models. Additionally, models trained on chip are inherently robust to hardware variations, allowing direct mapping to CIM inference chips without additional re-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_14547 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Bulk-Switching Memristor-based Compute-In-Memory Module for Deep Neural Network Training Wu, Yuting Wang, Qiwen Wang, Ziyu Wang, Xinxin Ayyagari, Buvna Krishnan, Siddarth Chudzik, Michael Lu, Wei D. Hardware Architecture Emerging Technologies Machine Learning The need for deep neural network (DNN) models with higher performance and better functionality leads to the proliferation of very large models. Model training, however, requires intensive computation time and energy. Memristor-based compute-in-memory (CIM) modules can perform vector-matrix multiplication (VMM) in situ and in parallel, and have shown great promises in DNN inference applications. However, CIM-based model training faces challenges due to non-linear weight updates, device variations, and low-precision in analog computing circuits. In this work, we experimentally implement a mixed-precision training scheme to mitigate these effects using a bulk-switching memristor CIM module. Lowprecision CIM modules are used to accelerate the expensive VMM operations, with high precision weight updates accumulated in digital units. Memristor devices are only changed when the accumulated weight update value exceeds a pre-defined threshold. The proposed scheme is implemented with a system-on-chip (SoC) of fully integrated analog CIM modules and digital sub-systems, showing fast convergence of LeNet training to 97.73%. The efficacy of training larger models is evaluated using realistic hardware parameters and shows that that analog CIM modules can enable efficient mix-precision DNN training with accuracy comparable to full-precision software trained models. Additionally, models trained on chip are inherently robust to hardware variations, allowing direct mapping to CIM inference chips without additional re-training. |
| title | Bulk-Switching Memristor-based Compute-In-Memory Module for Deep Neural Network Training |
| topic | Hardware Architecture Emerging Technologies Machine Learning |
| url | https://arxiv.org/abs/2305.14547 |