Language Modeling on a SpiNNaker 2 Neuromorphic Chip
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914650495385600 |
|---|---|
| author | Nazeer, Khaleelulla Khan Schöne, Mark Mukherji, Rishav Vogginger, Bernhard Mayr, Christian Kappel, David Subramoney, Anand |
| author_facet | Nazeer, Khaleelulla Khan Schöne, Mark Mukherji, Rishav Vogginger, Bernhard Mayr, Christian Kappel, David Subramoney, Anand |
| contents | As large language models continue to scale in size rapidly, so too does the computational power required to run them. Event-based networks on neuromorphic devices offer a potential way to reduce energy consumption for inference significantly. However, to date, most event-based networks that can run on neuromorphic hardware, including spiking neural networks (SNNs), have not achieved task performance even on par with LSTM models for language modeling. As a result, language modeling on neuromorphic devices has seemed a distant prospect. In this work, we demonstrate the first-ever implementation of a language model on a neuromorphic device - specifically the SpiNNaker 2 chip - based on a recently published event-based architecture called the EGRU. SpiNNaker 2 is a many-core neuromorphic chip designed for large-scale asynchronous processing, while the EGRU is architected to leverage such hardware efficiently while maintaining competitive task performance. This implementation marks the first time a neuromorphic language model matches LSTMs, setting the stage for taking task performance to the level of large language models. We also demonstrate results on a gesture recognition task based on inputs from a DVS camera. Overall, our results showcase the feasibility of this neuro-inspired neural network in hardware, highlighting significant gains versus conventional hardware in energy efficiency for the common use case of single batch inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_09084 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Language Modeling on a SpiNNaker 2 Neuromorphic Chip Nazeer, Khaleelulla Khan Schöne, Mark Mukherji, Rishav Vogginger, Bernhard Mayr, Christian Kappel, David Subramoney, Anand Neural and Evolutionary Computing Computation and Language Emerging Technologies Machine Learning As large language models continue to scale in size rapidly, so too does the computational power required to run them. Event-based networks on neuromorphic devices offer a potential way to reduce energy consumption for inference significantly. However, to date, most event-based networks that can run on neuromorphic hardware, including spiking neural networks (SNNs), have not achieved task performance even on par with LSTM models for language modeling. As a result, language modeling on neuromorphic devices has seemed a distant prospect. In this work, we demonstrate the first-ever implementation of a language model on a neuromorphic device - specifically the SpiNNaker 2 chip - based on a recently published event-based architecture called the EGRU. SpiNNaker 2 is a many-core neuromorphic chip designed for large-scale asynchronous processing, while the EGRU is architected to leverage such hardware efficiently while maintaining competitive task performance. This implementation marks the first time a neuromorphic language model matches LSTMs, setting the stage for taking task performance to the level of large language models. We also demonstrate results on a gesture recognition task based on inputs from a DVS camera. Overall, our results showcase the feasibility of this neuro-inspired neural network in hardware, highlighting significant gains versus conventional hardware in energy efficiency for the common use case of single batch inference. |
| title | Language Modeling on a SpiNNaker 2 Neuromorphic Chip |
| topic | Neural and Evolutionary Computing Computation and Language Emerging Technologies Machine Learning |
| url | https://arxiv.org/abs/2312.09084 |