Mitigating Communication Costs in Neural Networks: The Role of Dendritic Nonlinearity
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916677118066688 |
|---|---|
| author | Wu, Xundong Zhao, Pengfei Yu, Zilin Ma, Lei Yip, Ka-Wa Tang, Huajin Pan, Gang Panayiota, Poirazi Huang, Tiejun |
| author_facet | Wu, Xundong Zhao, Pengfei Yu, Zilin Ma, Lei Yip, Ka-Wa Tang, Huajin Pan, Gang Panayiota, Poirazi Huang, Tiejun |
| contents | Our understanding of biological neuronal networks has profoundly influenced the development of artificial neural networks (ANNs). However, neurons utilized in ANNs differ considerably from their biological counterparts, primarily due to the absence of complex dendritic trees with local nonlinearities. Early studies have suggested that dendritic nonlinearities could substantially improve the learning capabilities of neural network models. In this study, we systematically examined the role of nonlinear dendrites within neural networks. Utilizing machine-learning methodologies, we assessed how dendritic nonlinearities influence neural network performance. Our findings demonstrate that dendritic nonlinearities do not substantially affect learning capacity; rather, their primary benefit lies in enabling network capacity expansion while minimizing communication costs through effective localized feature aggregation. This research provides critical insights with significant implications for designing future neural network accelerators aimed at reducing communication overhead during neural network training and inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_11950 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Mitigating Communication Costs in Neural Networks: The Role of Dendritic Nonlinearity Wu, Xundong Zhao, Pengfei Yu, Zilin Ma, Lei Yip, Ka-Wa Tang, Huajin Pan, Gang Panayiota, Poirazi Huang, Tiejun Neural and Evolutionary Computing Machine Learning Neurons and Cognition Our understanding of biological neuronal networks has profoundly influenced the development of artificial neural networks (ANNs). However, neurons utilized in ANNs differ considerably from their biological counterparts, primarily due to the absence of complex dendritic trees with local nonlinearities. Early studies have suggested that dendritic nonlinearities could substantially improve the learning capabilities of neural network models. In this study, we systematically examined the role of nonlinear dendrites within neural networks. Utilizing machine-learning methodologies, we assessed how dendritic nonlinearities influence neural network performance. Our findings demonstrate that dendritic nonlinearities do not substantially affect learning capacity; rather, their primary benefit lies in enabling network capacity expansion while minimizing communication costs through effective localized feature aggregation. This research provides critical insights with significant implications for designing future neural network accelerators aimed at reducing communication overhead during neural network training and inference. |
| title | Mitigating Communication Costs in Neural Networks: The Role of Dendritic Nonlinearity |
| topic | Neural and Evolutionary Computing Machine Learning Neurons and Cognition |
| url | https://arxiv.org/abs/2306.11950 |