Sampling the space of solutions of an artificial neural network
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912498795413504 |
|---|---|
| author | Zambon, Alessandro Malatesta, Enrico M. Tiana, Guido Zecchina, Riccardo |
| author_facet | Zambon, Alessandro Malatesta, Enrico M. Tiana, Guido Zecchina, Riccardo |
| contents | The weight space of an artificial neural network can be systematically explored using tools from statistical mechanics. We employ a combination of a hybrid Monte Carlo algorithm which performs long exploration steps, a ratchet-based algorithm to investigate connectivity paths, and coupled replica models simulations to study subdominant flat regions. Our analysis focuses on one hidden layer networks and spans a range of energy levels and constrained density regimes. Near the interpolation threshold, the low-energy manifold shows a spiky topology. In the overparameterized regime, however, the low-energy manifold becomes entirely flat, forming an extended complex structure that is easy to sample. These numerical results are supported by an analytical study of the training error landscape, and we show numerically that the qualitative features of the loss landscape are robust across different data structures.
Our study aims to provide new methodological insights for developing scalable methods for large networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_08266 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sampling the space of solutions of an artificial neural network Zambon, Alessandro Malatesta, Enrico M. Tiana, Guido Zecchina, Riccardo Disordered Systems and Neural Networks Statistical Mechanics Probability The weight space of an artificial neural network can be systematically explored using tools from statistical mechanics. We employ a combination of a hybrid Monte Carlo algorithm which performs long exploration steps, a ratchet-based algorithm to investigate connectivity paths, and coupled replica models simulations to study subdominant flat regions. Our analysis focuses on one hidden layer networks and spans a range of energy levels and constrained density regimes. Near the interpolation threshold, the low-energy manifold shows a spiky topology. In the overparameterized regime, however, the low-energy manifold becomes entirely flat, forming an extended complex structure that is easy to sample. These numerical results are supported by an analytical study of the training error landscape, and we show numerically that the qualitative features of the loss landscape are robust across different data structures. Our study aims to provide new methodological insights for developing scalable methods for large networks. |
| title | Sampling the space of solutions of an artificial neural network |
| topic | Disordered Systems and Neural Networks Statistical Mechanics Probability |
| url | https://arxiv.org/abs/2503.08266 |