Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929248456933376 |
|---|---|
| author | Agliari, Elena Alemanno, Francesco Aquaro, Miriam Fachechi, Alberto |
| author_facet | Agliari, Elena Alemanno, Francesco Aquaro, Miriam Fachechi, Alberto |
| contents | In this work we approach attractor neural networks from a machine learning perspective: we look for optimal network parameters by applying a gradient descent over a regularized loss function. Within this framework, the optimal neuron-interaction matrices turn out to be a class of matrices which correspond to Hebbian kernels revised by a reiterated unlearning protocol. Remarkably, the extent of such unlearning is proved to be related to the regularization hyperparameter of the loss function and to the training time. Thus, we can design strategies to avoid overfitting that are formulated in terms of regularization and early-stopping tuning. The generalization capabilities of these attractor networks are also investigated: analytical results are obtained for random synthetic datasets, next, the emerging picture is corroborated by numerical experiments that highlight the existence of several regimes (i.e., overfitting, failure and success) as the dataset parameters are varied. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_01421 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting Agliari, Elena Alemanno, Francesco Aquaro, Miriam Fachechi, Alberto Machine Learning Disordered Systems and Neural Networks In this work we approach attractor neural networks from a machine learning perspective: we look for optimal network parameters by applying a gradient descent over a regularized loss function. Within this framework, the optimal neuron-interaction matrices turn out to be a class of matrices which correspond to Hebbian kernels revised by a reiterated unlearning protocol. Remarkably, the extent of such unlearning is proved to be related to the regularization hyperparameter of the loss function and to the training time. Thus, we can design strategies to avoid overfitting that are formulated in terms of regularization and early-stopping tuning. The generalization capabilities of these attractor networks are also investigated: analytical results are obtained for random synthetic datasets, next, the emerging picture is corroborated by numerical experiments that highlight the existence of several regimes (i.e., overfitting, failure and success) as the dataset parameters are varied. |
| title | Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2308.01421 |