Emergent representations in networks trained with the Forward-Forward algorithm
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917972223721472 |
|---|---|
| author | Tosato, Niccolò Basile, Lorenzo Ballarin, Emanuele de Alteriis, Giuseppe Cazzaniga, Alberto Ansuini, Alessio |
| author_facet | Tosato, Niccolò Basile, Lorenzo Ballarin, Emanuele de Alteriis, Giuseppe Cazzaniga, Alberto Ansuini, Alessio |
| contents | The Backpropagation algorithm has often been criticised for its lack of biological realism. In an attempt to find a more biologically plausible alternative, the recently introduced Forward-Forward algorithm replaces the forward and backward passes of Backpropagation with two forward passes. In this work, we show that the internal representations obtained by the Forward-Forward algorithm can organise into category-specific ensembles exhibiting high sparsity -- composed of a low number of active units. This situation is reminiscent of what has been observed in cortical sensory areas, where neuronal ensembles are suggested to serve as the functional building blocks for perception and action. Interestingly, while this sparse pattern does not typically arise in models trained with standard Backpropagation, it can emerge in networks trained with Backpropagation on the same objective proposed for the Forward-Forward algorithm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_18353 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Emergent representations in networks trained with the Forward-Forward algorithm Tosato, Niccolò Basile, Lorenzo Ballarin, Emanuele de Alteriis, Giuseppe Cazzaniga, Alberto Ansuini, Alessio Neural and Evolutionary Computing Machine Learning The Backpropagation algorithm has often been criticised for its lack of biological realism. In an attempt to find a more biologically plausible alternative, the recently introduced Forward-Forward algorithm replaces the forward and backward passes of Backpropagation with two forward passes. In this work, we show that the internal representations obtained by the Forward-Forward algorithm can organise into category-specific ensembles exhibiting high sparsity -- composed of a low number of active units. This situation is reminiscent of what has been observed in cortical sensory areas, where neuronal ensembles are suggested to serve as the functional building blocks for perception and action. Interestingly, while this sparse pattern does not typically arise in models trained with standard Backpropagation, it can emerge in networks trained with Backpropagation on the same objective proposed for the Forward-Forward algorithm. |
| title | Emergent representations in networks trained with the Forward-Forward algorithm |
| topic | Neural and Evolutionary Computing Machine Learning |
| url | https://arxiv.org/abs/2305.18353 |