Generalization performance of narrow one-hidden layer networks in the teacher-student setting
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908913047175168 |
|---|---|
| author | Ortiz, Rodrigo Pérez Nwemadji, Gibbs Barbier, Jean Gerace, Federica Ingrosso, Alessandro Lauditi, Clarissa Malatesta, Enrico M. |
| author_facet | Ortiz, Rodrigo Pérez Nwemadji, Gibbs Barbier, Jean Gerace, Federica Ingrosso, Alessandro Lauditi, Clarissa Malatesta, Enrico M. |
| contents | Understanding the generalization properties of neural networks on simple input-output distributions is key to explaining their performance on real datasets. The classical teacher-student setting, where a network is trained on data generated by a teacher model, provides a canonical theoretical test bed. In this context, a complete theoretical characterization of fully connected one-hidden-layer networks with generic activation functions remains missing. In this work, we develop a general framework for such networks with large width, yet much smaller than the input dimension. Using methods from statistical physics, we derive closed-form expressions for the typical performance of both finite-temperature (Bayesian) and empirical risk minimization estimators in terms of a small number of order parameters. We uncover a transition to a specialization phase, where hidden neurons align with teacher features once the number of samples becomes sufficiently large and proportional to the number of network parameters. Our theory accurately predicts the generalization error of networks trained on regression and classification tasks using either noisy full-batch gradient descent (Langevin dynamics) or deterministic full-batch gradient descent. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_00629 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Generalization performance of narrow one-hidden layer networks in the teacher-student setting Ortiz, Rodrigo Pérez Nwemadji, Gibbs Barbier, Jean Gerace, Federica Ingrosso, Alessandro Lauditi, Clarissa Malatesta, Enrico M. Disordered Systems and Neural Networks Machine Learning Probability Statistics Theory Understanding the generalization properties of neural networks on simple input-output distributions is key to explaining their performance on real datasets. The classical teacher-student setting, where a network is trained on data generated by a teacher model, provides a canonical theoretical test bed. In this context, a complete theoretical characterization of fully connected one-hidden-layer networks with generic activation functions remains missing. In this work, we develop a general framework for such networks with large width, yet much smaller than the input dimension. Using methods from statistical physics, we derive closed-form expressions for the typical performance of both finite-temperature (Bayesian) and empirical risk minimization estimators in terms of a small number of order parameters. We uncover a transition to a specialization phase, where hidden neurons align with teacher features once the number of samples becomes sufficiently large and proportional to the number of network parameters. Our theory accurately predicts the generalization error of networks trained on regression and classification tasks using either noisy full-batch gradient descent (Langevin dynamics) or deterministic full-batch gradient descent. |
| title | Generalization performance of narrow one-hidden layer networks in the teacher-student setting |
| topic | Disordered Systems and Neural Networks Machine Learning Probability Statistics Theory |
| url | https://arxiv.org/abs/2507.00629 |