Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915125426913280 |
|---|---|
| author | Knunyants, Ivan Tavakol, Maryam Sifalakis, Manolis Xu, Yingfu Yousefzadeh, Amirreza Tang, Guangzhi |
| author_facet | Knunyants, Ivan Tavakol, Maryam Sifalakis, Manolis Xu, Yingfu Yousefzadeh, Amirreza Tang, Guangzhi |
| contents | The recent rise of Large Language Models (LLMs) has revolutionized the deep learning field. However, the desire to deploy LLMs on edge devices introduces energy efficiency and latency challenges. Recurrent LLM (R-LLM) architectures have proven effective in mitigating the quadratic complexity of self-attention, making them a potential paradigm for computing on-edge neuromorphic processors. In this work, we propose a low-cost, training-free algorithm to sparsify R-LLMs' activations to enhance energy efficiency on neuromorphic hardware. Our approach capitalizes on the inherent structure of these models, rendering them well-suited for energy-constrained environments. Although primarily designed for R-LLMs, this method can be generalized to other LLM architectures, such as transformers, as demonstrated on the OPT model, achieving comparable sparsity and efficiency improvements. Empirical studies illustrate that our method significantly reduces computational demands while maintaining competitive accuracy across multiple zero-shot learning benchmarks. Additionally, hardware simulations with the SENECA neuromorphic processor underscore notable energy savings and latency improvements. These results pave the way for low-power, real-time neuromorphic deployment of LLMs and demonstrate the feasibility of training-free on-chip adaptation using activation sparsity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16337 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing Knunyants, Ivan Tavakol, Maryam Sifalakis, Manolis Xu, Yingfu Yousefzadeh, Amirreza Tang, Guangzhi Neural and Evolutionary Computing Artificial Intelligence Hardware Architecture Machine Learning The recent rise of Large Language Models (LLMs) has revolutionized the deep learning field. However, the desire to deploy LLMs on edge devices introduces energy efficiency and latency challenges. Recurrent LLM (R-LLM) architectures have proven effective in mitigating the quadratic complexity of self-attention, making them a potential paradigm for computing on-edge neuromorphic processors. In this work, we propose a low-cost, training-free algorithm to sparsify R-LLMs' activations to enhance energy efficiency on neuromorphic hardware. Our approach capitalizes on the inherent structure of these models, rendering them well-suited for energy-constrained environments. Although primarily designed for R-LLMs, this method can be generalized to other LLM architectures, such as transformers, as demonstrated on the OPT model, achieving comparable sparsity and efficiency improvements. Empirical studies illustrate that our method significantly reduces computational demands while maintaining competitive accuracy across multiple zero-shot learning benchmarks. Additionally, hardware simulations with the SENECA neuromorphic processor underscore notable energy savings and latency improvements. These results pave the way for low-power, real-time neuromorphic deployment of LLMs and demonstrate the feasibility of training-free on-chip adaptation using activation sparsity. |
| title | Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing |
| topic | Neural and Evolutionary Computing Artificial Intelligence Hardware Architecture Machine Learning |
| url | https://arxiv.org/abs/2501.16337 |