Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909560624644096 |
|---|---|
| author | Barbier, Jean Camilli, Francesco Nguyen, Minh-Toan Pastore, Mauro Skerk, Rudy |
| author_facet | Barbier, Jean Camilli, Francesco Nguyen, Minh-Toan Pastore, Mauro Skerk, Rudy |
| contents | We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network whose width $k$ and input dimension $d$ are large and proportional. We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function in the regime of sample size $n$ scaling quadratically with the input dimension, i.e., around the interpolation threshold where the number of trainable parameters $kd+k$ and of data $n$ are comparable. Our analysis tackles generic weight distributions. We uncover a discontinuous phase transition separating a "universal" phase from a "specialisation" phase. In the first, the generalisation error is independent of the weight distribution and decays slowly with the sampling rate $n/d^2$, with the student learning only some non-linear combinations of the teacher weights. In the latter, the error is weight distribution-dependent and decays faster due to the alignment of the student towards the teacher network. We thus unveil the existence of a highly predictive solution near interpolation, which is however potentially hard to find by practical algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_18530 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation Barbier, Jean Camilli, Francesco Nguyen, Minh-Toan Pastore, Mauro Skerk, Rudy Machine Learning Disordered Systems and Neural Networks Statistical Mechanics Information Theory We consider a teacher-student model of supervised learning with a fully-trained two-layer neural network whose width $k$ and input dimension $d$ are large and proportional. We provide an effective theory for approximating the Bayes-optimal generalisation error of the network for any activation function in the regime of sample size $n$ scaling quadratically with the input dimension, i.e., around the interpolation threshold where the number of trainable parameters $kd+k$ and of data $n$ are comparable. Our analysis tackles generic weight distributions. We uncover a discontinuous phase transition separating a "universal" phase from a "specialisation" phase. In the first, the generalisation error is independent of the weight distribution and decays slowly with the sampling rate $n/d^2$, with the student learning only some non-linear combinations of the teacher weights. In the latter, the error is weight distribution-dependent and decays faster due to the alignment of the student towards the teacher network. We thus unveil the existence of a highly predictive solution near interpolation, which is however potentially hard to find by practical algorithms. |
| title | Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation |
| topic | Machine Learning Disordered Systems and Neural Networks Statistical Mechanics Information Theory |
| url | https://arxiv.org/abs/2501.18530 |