A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2022
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910320238264320 |
|---|---|
| author | Salvatori, Tommaso Song, Yuhang Yordanov, Yordan Millidge, Beren Xu, Zhenghua Sha, Lei Emde, Cornelius Bogacz, Rafal Lukasiewicz, Thomas |
| author_facet | Salvatori, Tommaso Song, Yuhang Yordanov, Yordan Millidge, Beren Xu, Zhenghua Sha, Lei Emde, Cornelius Bogacz, Rafal Lukasiewicz, Thomas |
| contents | Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. Training such models, however, is quite inefficient and unstable. In this work, we show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one, and has theoretical guarantees in terms of convergence. The proposed algorithm, that we call incremental predictive coding (iPC) is also more biologically plausible than the original one, as it it fully automatic. In an extensive set of experiments, we show that iPC constantly performs better than the original formulation on a large number of benchmarks for image classification, as well as for the training of both conditional and masked language models, in terms of test accuracy, efficiency, and convergence with respect to a large set of hyperparameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2212_00720 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks Salvatori, Tommaso Song, Yuhang Yordanov, Yordan Millidge, Beren Xu, Zhenghua Sha, Lei Emde, Cornelius Bogacz, Rafal Lukasiewicz, Thomas Neural and Evolutionary Computing Artificial Intelligence Machine Learning Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. Training such models, however, is quite inefficient and unstable. In this work, we show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one, and has theoretical guarantees in terms of convergence. The proposed algorithm, that we call incremental predictive coding (iPC) is also more biologically plausible than the original one, as it it fully automatic. In an extensive set of experiments, we show that iPC constantly performs better than the original formulation on a large number of benchmarks for image classification, as well as for the training of both conditional and masked language models, in terms of test accuracy, efficiency, and convergence with respect to a large set of hyperparameters. |
| title | A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks |
| topic | Neural and Evolutionary Computing Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2212.00720 |