Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2409.18683 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916412696559616 |
|---|---|
| author | Mele, Margherita Menichetti, Roberto Ingrosso, Alessandro Potestio, Raffaello |
| author_facet | Mele, Margherita Menichetti, Roberto Ingrosso, Alessandro Potestio, Raffaello |
| contents | Learning in neural networks critically hinges on the intricate geometry of the loss landscape associated with a given task. Traditionally, most research has focused on finding specific weight configurations that minimize the loss. In this work, born from the cross-fertilization of machine learning and theoretical soft matter physics, we introduce a novel, computationally efficient approach to examine the weight space across all loss values. Employing the Wang-Landau enhanced sampling algorithm, we explore the neural network density of states - the number of network parameter configurations that produce a given loss value - and analyze how it depends on specific features of the training set. Using both real-world and synthetic data, we quantitatively elucidate the relation between data structure and network density of states across different sizes and depths of binary-state networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18683 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Density of states in neural networks: an in-depth exploration of learning in parameter space Mele, Margherita Menichetti, Roberto Ingrosso, Alessandro Potestio, Raffaello Statistical Mechanics Learning in neural networks critically hinges on the intricate geometry of the loss landscape associated with a given task. Traditionally, most research has focused on finding specific weight configurations that minimize the loss. In this work, born from the cross-fertilization of machine learning and theoretical soft matter physics, we introduce a novel, computationally efficient approach to examine the weight space across all loss values. Employing the Wang-Landau enhanced sampling algorithm, we explore the neural network density of states - the number of network parameter configurations that produce a given loss value - and analyze how it depends on specific features of the training set. Using both real-world and synthetic data, we quantitatively elucidate the relation between data structure and network density of states across different sizes and depths of binary-state networks. |
| title | Density of states in neural networks: an in-depth exploration of learning in parameter space |
| topic | Statistical Mechanics |
| url | https://arxiv.org/abs/2409.18683 |