Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929587595771904 |
|---|---|
| author | Wan, Ruyin Zhang, Qian Karniadakis, George Em |
| author_facet | Wan, Ruyin Zhang, Qian Karniadakis, George Em |
| contents | Spiking neural networks (SNNs) represent a promising approach in machine learning, combining the hierarchical learning capabilities of deep neural networks with the energy efficiency of spike-based computations. Traditional end-to-end training of SNNs is often based on back-propagation, where weight updates are derived from gradients computed through the chain rule. However, this method encounters challenges due to its limited biological plausibility and inefficiencies on neuromorphic hardware. In this study, we introduce an alternative training approach for SNNs. Instead of using back-propagation, we leverage weight perturbation methods within a forward-mode gradient framework. Specifically, we perturb the weight matrix with a small noise term and estimate gradients by observing the changes in the network output. Experimental results on regression tasks, including solving various PDEs, show that our approach achieves competitive accuracy, suggesting its suitability for neuromorphic systems and potential hardware compatibility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07057 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning Wan, Ruyin Zhang, Qian Karniadakis, George Em Neural and Evolutionary Computing Numerical Analysis Optimization and Control Spiking neural networks (SNNs) represent a promising approach in machine learning, combining the hierarchical learning capabilities of deep neural networks with the energy efficiency of spike-based computations. Traditional end-to-end training of SNNs is often based on back-propagation, where weight updates are derived from gradients computed through the chain rule. However, this method encounters challenges due to its limited biological plausibility and inefficiencies on neuromorphic hardware. In this study, we introduce an alternative training approach for SNNs. Instead of using back-propagation, we leverage weight perturbation methods within a forward-mode gradient framework. Specifically, we perturb the weight matrix with a small noise term and estimate gradients by observing the changes in the network output. Experimental results on regression tasks, including solving various PDEs, show that our approach achieves competitive accuracy, suggesting its suitability for neuromorphic systems and potential hardware compatibility. |
| title | Randomized Forward Mode Gradient for Spiking Neural Networks in Scientific Machine Learning |
| topic | Neural and Evolutionary Computing Numerical Analysis Optimization and Control |
| url | https://arxiv.org/abs/2411.07057 |