Expressivity of Neural Networks with Random Weights and Learned Biases
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909547639078912 |
|---|---|
| author | Williams, Ezekiel Payeur, Alexandre Ryoo, Avery Hee-Woon Jiralerspong, Thomas Perich, Matthew G. Mazzucato, Luca Lajoie, Guillaume |
| author_facet | Williams, Ezekiel Payeur, Alexandre Ryoo, Avery Hee-Woon Jiralerspong, Thomas Perich, Matthew G. Mazzucato, Luca Lajoie, Guillaume |
| contents | Landmark universal function approximation results for neural networks with trained weights and biases provided the impetus for the ubiquitous use of neural networks as learning models in neuroscience and Artificial Intelligence (AI). Recent work has extended these results to networks in which a smaller subset of weights (e.g., output weights) are tuned, leaving other parameters random. However, it remains an open question whether universal approximation holds when only biases are learned, despite evidence from neuroscience and AI that biases significantly shape neural responses. The current paper answers this question. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can approximate any continuous function on compact sets. We further show an analogous result for the approximation of dynamical systems with recurrent neural networks. Our findings are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights, as well as for AI, where they shed light on recent fine-tuning methods for large language models, like bias and prefix-based approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00957 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Expressivity of Neural Networks with Random Weights and Learned Biases Williams, Ezekiel Payeur, Alexandre Ryoo, Avery Hee-Woon Jiralerspong, Thomas Perich, Matthew G. Mazzucato, Luca Lajoie, Guillaume Neural and Evolutionary Computing Neurons and Cognition Machine Learning Landmark universal function approximation results for neural networks with trained weights and biases provided the impetus for the ubiquitous use of neural networks as learning models in neuroscience and Artificial Intelligence (AI). Recent work has extended these results to networks in which a smaller subset of weights (e.g., output weights) are tuned, leaving other parameters random. However, it remains an open question whether universal approximation holds when only biases are learned, despite evidence from neuroscience and AI that biases significantly shape neural responses. The current paper answers this question. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can approximate any continuous function on compact sets. We further show an analogous result for the approximation of dynamical systems with recurrent neural networks. Our findings are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights, as well as for AI, where they shed light on recent fine-tuning methods for large language models, like bias and prefix-based approaches. |
| title | Expressivity of Neural Networks with Random Weights and Learned Biases |
| topic | Neural and Evolutionary Computing Neurons and Cognition Machine Learning |
| url | https://arxiv.org/abs/2407.00957 |