Asymptotics of feature learning in two-layer networks after one gradient-step
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912014917435392 |
|---|---|
| author | Cui, Hugo Pesce, Luca Dandi, Yatin Krzakala, Florent Lu, Yue M. Zdeborová, Lenka Loureiro, Bruno |
| author_facet | Cui, Hugo Pesce, Luca Dandi, Yatin Krzakala, Florent Lu, Yue M. Zdeborová, Lenka Loureiro, Bruno |
| contents | In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step. Leveraging the insight from (Ba et al., 2022), we model the trained network by a spiked Random Features (sRF) model. Further building on recent progress on Gaussian universality (Dandi et al., 2023), we provide an exact asymptotic description of the generalization error of the sRF in the high-dimensional limit where the number of samples, the width, and the input dimension grow at a proportional rate. The resulting characterization for sRFs also captures closely the learning curves of the original network model. This enables us to understand how adapting to the data is crucial for the network to efficiently learn non-linear functions in the direction of the gradient -- where at initialization it can only express linear functions in this regime. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_04980 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Asymptotics of feature learning in two-layer networks after one gradient-step Cui, Hugo Pesce, Luca Dandi, Yatin Krzakala, Florent Lu, Yue M. Zdeborová, Lenka Loureiro, Bruno Machine Learning Disordered Systems and Neural Networks In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step. Leveraging the insight from (Ba et al., 2022), we model the trained network by a spiked Random Features (sRF) model. Further building on recent progress on Gaussian universality (Dandi et al., 2023), we provide an exact asymptotic description of the generalization error of the sRF in the high-dimensional limit where the number of samples, the width, and the input dimension grow at a proportional rate. The resulting characterization for sRFs also captures closely the learning curves of the original network model. This enables us to understand how adapting to the data is crucial for the network to efficiently learn non-linear functions in the direction of the gradient -- where at initialization it can only express linear functions in this regime. |
| title | Asymptotics of feature learning in two-layer networks after one gradient-step |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2402.04980 |