Stochastic Spiking Attention: Accelerating Attention with Stochastic Computing in Spiking Networks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929584475209728 |
|---|---|
| author | Song, Zihang Katti, Prabodh Simeone, Osvaldo Rajendran, Bipin |
| author_facet | Song, Zihang Katti, Prabodh Simeone, Osvaldo Rajendran, Bipin |
| contents | Spiking Neural Networks (SNNs) have been recently integrated into Transformer architectures due to their potential to reduce computational demands and to improve power efficiency. Yet, the implementation of the attention mechanism using spiking signals on general-purpose computing platforms remains inefficient. In this paper, we propose a novel framework leveraging stochastic computing (SC) to effectively execute the dot-product attention for SNN-based Transformers. We demonstrate that our approach can achieve high classification accuracy ($83.53\%$) on CIFAR-10 within 10 time steps, which is comparable to the performance of a baseline artificial neural network implementation ($83.66\%$). We estimate that the proposed SC approach can lead to over $6.3\times$ reduction in computing energy and $1.7\times$ reduction in memory access costs for a digital CMOS-based ASIC design. We experimentally validate our stochastic attention block design through an FPGA implementation, which is shown to achieve $48\times$ lower latency as compared to a GPU implementation, while consuming $15\times$ less power. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_09109 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Stochastic Spiking Attention: Accelerating Attention with Stochastic Computing in Spiking Networks Song, Zihang Katti, Prabodh Simeone, Osvaldo Rajendran, Bipin Hardware Architecture Artificial Intelligence Machine Learning Neural and Evolutionary Computing Signal Processing Spiking Neural Networks (SNNs) have been recently integrated into Transformer architectures due to their potential to reduce computational demands and to improve power efficiency. Yet, the implementation of the attention mechanism using spiking signals on general-purpose computing platforms remains inefficient. In this paper, we propose a novel framework leveraging stochastic computing (SC) to effectively execute the dot-product attention for SNN-based Transformers. We demonstrate that our approach can achieve high classification accuracy ($83.53\%$) on CIFAR-10 within 10 time steps, which is comparable to the performance of a baseline artificial neural network implementation ($83.66\%$). We estimate that the proposed SC approach can lead to over $6.3\times$ reduction in computing energy and $1.7\times$ reduction in memory access costs for a digital CMOS-based ASIC design. We experimentally validate our stochastic attention block design through an FPGA implementation, which is shown to achieve $48\times$ lower latency as compared to a GPU implementation, while consuming $15\times$ less power. |
| title | Stochastic Spiking Attention: Accelerating Attention with Stochastic Computing in Spiking Networks |
| topic | Hardware Architecture Artificial Intelligence Machine Learning Neural and Evolutionary Computing Signal Processing |
| url | https://arxiv.org/abs/2402.09109 |