A More Accurate Approximation of Activation Function with Few Spikes Neurons
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914931492782080 |
|---|---|
| author | Jeong, Dayena Park, Jaewoo Jo, Jeonghee Park, Jongkil Kim, Jaewook Jang, Hyun Jae Lee, Suyoun Park, Seongsik |
| author_facet | Jeong, Dayena Park, Jaewoo Jo, Jeonghee Park, Jongkil Kim, Jaewook Jang, Hyun Jae Lee, Suyoun Park, Seongsik |
| contents | Recent deep neural networks (DNNs), such as diffusion models [1], have faced high computational demands. Thus, spiking neural networks (SNNs) have attracted lots of attention as energy-efficient neural networks. However, conventional spiking neurons, such as leaky integrate-and-fire neurons, cannot accurately represent complex non-linear activation functions, such as Swish [2]. To approximate activation functions with spiking neurons, few spikes (FS) neurons were proposed [3], but the approximation performance was limited due to the lack of training methods considering the neurons. Thus, we propose tendency-based parameter initialization (TBPI) to enhance the approximation of activation function with FS neurons, exploiting temporal dependencies initializing the training parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00044 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A More Accurate Approximation of Activation Function with Few Spikes Neurons Jeong, Dayena Park, Jaewoo Jo, Jeonghee Park, Jongkil Kim, Jaewook Jang, Hyun Jae Lee, Suyoun Park, Seongsik Neural and Evolutionary Computing Machine Learning Recent deep neural networks (DNNs), such as diffusion models [1], have faced high computational demands. Thus, spiking neural networks (SNNs) have attracted lots of attention as energy-efficient neural networks. However, conventional spiking neurons, such as leaky integrate-and-fire neurons, cannot accurately represent complex non-linear activation functions, such as Swish [2]. To approximate activation functions with spiking neurons, few spikes (FS) neurons were proposed [3], but the approximation performance was limited due to the lack of training methods considering the neurons. Thus, we propose tendency-based parameter initialization (TBPI) to enhance the approximation of activation function with FS neurons, exploiting temporal dependencies initializing the training parameters. |
| title | A More Accurate Approximation of Activation Function with Few Spikes Neurons |
| topic | Neural and Evolutionary Computing Machine Learning |
| url | https://arxiv.org/abs/2409.00044 |