Resistive memory-based zero-shot liquid state machine for multimodal event data learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929668892917760 |
|---|---|
| author | Lin, Ning Wang, Shaocong Li, Yi Wang, Bo Shi, Shuhui He, Yangu Zhang, Woyu Yu, Yifei Zhang, Yue Zhang, Xinyuan Wong, Kwunhang Wang, Songqi Chen, Xiaoming Jiang, Hao Zhang, Xumeng Lin, Peng Xu, Xiaoxin Qi, Xiaojuan Wang, Zhongrui Shang, Dashan Liu, Qi Liu, Ming |
| author_facet | Lin, Ning Wang, Shaocong Li, Yi Wang, Bo Shi, Shuhui He, Yangu Zhang, Woyu Yu, Yifei Zhang, Yue Zhang, Xinyuan Wong, Kwunhang Wang, Songqi Chen, Xiaoming Jiang, Hao Zhang, Xumeng Lin, Peng Xu, Xiaoxin Qi, Xiaojuan Wang, Zhongrui Shang, Dashan Liu, Qi Liu, Ming |
| contents | The human brain is a complex spiking neural network (SNN), capable of learning multimodal signals in a zero-shot manner by generalizing existing knowledge. Remarkably, it maintains minimal power consumption through event-based signal propagation. However, replicating the human brain in neuromorphic hardware presents both hardware and software challenges. Hardware limitations, such as the slowdown of Moore's law and Von Neumann bottleneck, hinder the efficiency of digital computers. Additionally, SNNs are characterized by their software training complexities. To this end, we propose a hardware-software co-design on a 40 nm 256 Kb in-memory computing macro that physically integrates a fixed and random liquid state machine (LSM) SNN encoder with trainable artificial neural network (ANN) projections. We showcase the zero-shot LSM-based learning of multimodal events on the N-MNIST and N-TIDIGITS datasets, including visual and audio data association, as well as neural and visual data alignment for brain-machine interfaces. Our co-design achieves classification accuracy comparable to fully optimized software models, resulting in a 152.83 and 393.07-fold reduction in training costs compared to SOTA contrastive language-image pre-training (CLIP) and Prototypical networks, and a 23.34 and 160-fold improvement in energy efficiency compared to cutting-edge digital hardware, respectively. These proof-of-principle prototypes demonstrate zero-shot multimodal events learning capability for emerging efficient and compact neuromorphic hardware. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_00771 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Resistive memory-based zero-shot liquid state machine for multimodal event data learning Lin, Ning Wang, Shaocong Li, Yi Wang, Bo Shi, Shuhui He, Yangu Zhang, Woyu Yu, Yifei Zhang, Yue Zhang, Xinyuan Wong, Kwunhang Wang, Songqi Chen, Xiaoming Jiang, Hao Zhang, Xumeng Lin, Peng Xu, Xiaoxin Qi, Xiaojuan Wang, Zhongrui Shang, Dashan Liu, Qi Liu, Ming Emerging Technologies The human brain is a complex spiking neural network (SNN), capable of learning multimodal signals in a zero-shot manner by generalizing existing knowledge. Remarkably, it maintains minimal power consumption through event-based signal propagation. However, replicating the human brain in neuromorphic hardware presents both hardware and software challenges. Hardware limitations, such as the slowdown of Moore's law and Von Neumann bottleneck, hinder the efficiency of digital computers. Additionally, SNNs are characterized by their software training complexities. To this end, we propose a hardware-software co-design on a 40 nm 256 Kb in-memory computing macro that physically integrates a fixed and random liquid state machine (LSM) SNN encoder with trainable artificial neural network (ANN) projections. We showcase the zero-shot LSM-based learning of multimodal events on the N-MNIST and N-TIDIGITS datasets, including visual and audio data association, as well as neural and visual data alignment for brain-machine interfaces. Our co-design achieves classification accuracy comparable to fully optimized software models, resulting in a 152.83 and 393.07-fold reduction in training costs compared to SOTA contrastive language-image pre-training (CLIP) and Prototypical networks, and a 23.34 and 160-fold improvement in energy efficiency compared to cutting-edge digital hardware, respectively. These proof-of-principle prototypes demonstrate zero-shot multimodal events learning capability for emerging efficient and compact neuromorphic hardware. |
| title | Resistive memory-based zero-shot liquid state machine for multimodal event data learning |
| topic | Emerging Technologies |
| url | https://arxiv.org/abs/2307.00771 |