Saved in:
| Main Author: | |
|---|---|
| Format: | Recurso digital |
| Language: | |
| Published: |
Zenodo
2023
|
| Online Access: | https://doi.org/10.5281/zenodo.10620442 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866901655511891968 |
|---|---|
| author | Heckel, Kade |
| author_facet | Heckel, Kade |
| contents | <div> <div><span>While incredible achievements have been attained using 2nd generation artificial neural networks</span><br><span>(ANN) in the past few years, the computational and financial resources required to train and deploy</span><br><span>models at scale pose a significant challenge.</span> <span>While endeavors like knowledge distillation, weight</span><br><span>pruning, and quantization attempt to reduce computational demand in ANN architectures, they often</span><br><span>lead to significant quality loss for smaller models in resource-constrained setups; in contrast, the</span><br><span>emerging field of neuromorphic computing addresses these issues by adapting hardware and net-</span><br><span>work structures using energy-efficient principles inspired by biology. Using the brain as a blueprint,</span><br><span>neuromorphic systems depart from ANN architectures by limiting neurons to only communicate via</span><br><span>binary spikes; such a construction allows for the elimination of multiplication of floating point val-</span><br><span>ues and opens the door for asynchronous and event-triggered computation, vastly reducing the</span><br><span>complexity and power demands of hardware.</span> <span>These benefits are not reaped for free however as</span><br><span>the all-or-nothing firing in spiking neural networks (SNN) introduces new challenges in the training</span><br><span>process as their discontinuous dynamics yield an ill-defined gradient when performing optimization.</span><br><span>Hailed as the 3rd generation of neural computation, recent SNN optimization has been facilitated by</span><br><span>surrogate gradient methods which perform Back Propagation Through Time (BPTT) with a smoothed</span><br><span>approximation of the spiking activation function’s gradient. However, while surrogate gradients ad-</span><br><span>dress the concern that exact gradient methods are ignorant of spike generation or deletion, they</span><br><span>still face the issue of dead neurons which once completely silent cannot trigger weight updates.</span><br><span>Neuroevolution, a gradient-free optimization approach that had fallen out of favor until recent years</span><br><span>due computational constraints, provides a resolution to these challenges by circumventing the non-</span><br><span>differentiability of spiking activation functions.</span> <span>This thesis brings two distinct contributions: firstly,</span><br><span>the introduction of a new framework called Spyx, which enhances SNN training speed by an or-</span><br><span>der of magnitude, and secondly, an exploration of modern neuroevolution’s effectiveness for SNNs,</span><br><span>demonstrating competitive performance with gradient-based methods across multiple benchmark</span><br><span>datasets</span></div> </div> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_10620442 |
| institution | Zenodo |
| language | |
| publishDate | 2023 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Neuroevolution of Spiking Neural Networks Heckel, Kade <div> <div><span>While incredible achievements have been attained using 2nd generation artificial neural networks</span><br><span>(ANN) in the past few years, the computational and financial resources required to train and deploy</span><br><span>models at scale pose a significant challenge.</span> <span>While endeavors like knowledge distillation, weight</span><br><span>pruning, and quantization attempt to reduce computational demand in ANN architectures, they often</span><br><span>lead to significant quality loss for smaller models in resource-constrained setups; in contrast, the</span><br><span>emerging field of neuromorphic computing addresses these issues by adapting hardware and net-</span><br><span>work structures using energy-efficient principles inspired by biology. Using the brain as a blueprint,</span><br><span>neuromorphic systems depart from ANN architectures by limiting neurons to only communicate via</span><br><span>binary spikes; such a construction allows for the elimination of multiplication of floating point val-</span><br><span>ues and opens the door for asynchronous and event-triggered computation, vastly reducing the</span><br><span>complexity and power demands of hardware.</span> <span>These benefits are not reaped for free however as</span><br><span>the all-or-nothing firing in spiking neural networks (SNN) introduces new challenges in the training</span><br><span>process as their discontinuous dynamics yield an ill-defined gradient when performing optimization.</span><br><span>Hailed as the 3rd generation of neural computation, recent SNN optimization has been facilitated by</span><br><span>surrogate gradient methods which perform Back Propagation Through Time (BPTT) with a smoothed</span><br><span>approximation of the spiking activation function’s gradient. However, while surrogate gradients ad-</span><br><span>dress the concern that exact gradient methods are ignorant of spike generation or deletion, they</span><br><span>still face the issue of dead neurons which once completely silent cannot trigger weight updates.</span><br><span>Neuroevolution, a gradient-free optimization approach that had fallen out of favor until recent years</span><br><span>due computational constraints, provides a resolution to these challenges by circumventing the non-</span><br><span>differentiability of spiking activation functions.</span> <span>This thesis brings two distinct contributions: firstly,</span><br><span>the introduction of a new framework called Spyx, which enhances SNN training speed by an or-</span><br><span>der of magnitude, and secondly, an exploration of modern neuroevolution’s effectiveness for SNNs,</span><br><span>demonstrating competitive performance with gradient-based methods across multiple benchmark</span><br><span>datasets</span></div> </div> |
| title | Neuroevolution of Spiking Neural Networks |
| url | https://doi.org/10.5281/zenodo.10620442 |