Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2412.05540 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910732997623808 |
|---|---|
| author | Xu, Boxun Hwang, Junyoung Vanna-iampikul, Pruek Yin, Yuxuan Lim, Sung Kyu Li, Peng |
| author_facet | Xu, Boxun Hwang, Junyoung Vanna-iampikul, Pruek Yin, Yuxuan Lim, Sung Kyu Li, Peng |
| contents | Spiking Neural Networks(SNNs) provide a brain-inspired and event-driven mechanism that is believed to be critical to unlock energy-efficient deep learning. The mixture-of-experts approach mirrors the parallel distributed processing of nervous systems, introducing conditional computation policies and expanding model capacity without scaling up the number of computational operations. Additionally, spiking mixture-of-experts self-attention mechanisms enhance representation capacity, effectively capturing diverse patterns of entities and dependencies between visual or linguistic tokens. However, there is currently a lack of hardware support for highly parallel distributed processing needed by spiking transformers, which embody a brain-inspired computation. This paper introduces the first 3D hardware architecture and design methodology for Mixture-of-Experts and Multi-Head Attention spiking transformers. By leveraging 3D integration with memory-on-logic and logic-on-logic stacking, we explore such brain-inspired accelerators with spatially stackable circuitry, demonstrating significant optimization of energy efficiency and latency compared to conventional 2D CMOS integration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_05540 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards 3D Acceleration for low-power Mixture-of-Experts and Multi-Head Attention Spiking Transformers Xu, Boxun Hwang, Junyoung Vanna-iampikul, Pruek Yin, Yuxuan Lim, Sung Kyu Li, Peng Neural and Evolutionary Computing Artificial Intelligence Hardware Architecture Spiking Neural Networks(SNNs) provide a brain-inspired and event-driven mechanism that is believed to be critical to unlock energy-efficient deep learning. The mixture-of-experts approach mirrors the parallel distributed processing of nervous systems, introducing conditional computation policies and expanding model capacity without scaling up the number of computational operations. Additionally, spiking mixture-of-experts self-attention mechanisms enhance representation capacity, effectively capturing diverse patterns of entities and dependencies between visual or linguistic tokens. However, there is currently a lack of hardware support for highly parallel distributed processing needed by spiking transformers, which embody a brain-inspired computation. This paper introduces the first 3D hardware architecture and design methodology for Mixture-of-Experts and Multi-Head Attention spiking transformers. By leveraging 3D integration with memory-on-logic and logic-on-logic stacking, we explore such brain-inspired accelerators with spatially stackable circuitry, demonstrating significant optimization of energy efficiency and latency compared to conventional 2D CMOS integration. |
| title | Towards 3D Acceleration for low-power Mixture-of-Experts and Multi-Head Attention Spiking Transformers |
| topic | Neural and Evolutionary Computing Artificial Intelligence Hardware Architecture |
| url | https://arxiv.org/abs/2412.05540 |