Statistical mechanics of transfer learning in fully-connected networks in the proportional limit
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916318173724672 |
|---|---|
| author | Ingrosso, Alessandro Pacelli, Rosalba Rotondo, Pietro Gerace, Federica |
| author_facet | Ingrosso, Alessandro Pacelli, Rosalba Rotondo, Pietro Gerace, Federica |
| contents | Transfer learning (TL) is a well-established machine learning technique to boost the generalization performance on a specific (target) task using information gained from a related (source) task, and it crucially depends on the ability of a network to learn useful features. Leveraging recent analytical progress in the proportional regime of deep learning theory (i.e. the limit where the size of the training set $P$ and the size of the hidden layers $N$ are taken to infinity keeping their ratio $α= P/N$ finite), in this work we develop a novel single-instance Franz-Parisi formalism that yields an effective theory for TL in fully-connected neural networks. Unlike the (lazy-training) infinite-width limit, where TL is ineffective, we demonstrate that in the proportional limit TL occurs due to a renormalized source-target kernel that quantifies their relatedness and determines whether TL is beneficial for generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07168 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Statistical mechanics of transfer learning in fully-connected networks in the proportional limit Ingrosso, Alessandro Pacelli, Rosalba Rotondo, Pietro Gerace, Federica Disordered Systems and Neural Networks Statistical Mechanics Transfer learning (TL) is a well-established machine learning technique to boost the generalization performance on a specific (target) task using information gained from a related (source) task, and it crucially depends on the ability of a network to learn useful features. Leveraging recent analytical progress in the proportional regime of deep learning theory (i.e. the limit where the size of the training set $P$ and the size of the hidden layers $N$ are taken to infinity keeping their ratio $α= P/N$ finite), in this work we develop a novel single-instance Franz-Parisi formalism that yields an effective theory for TL in fully-connected neural networks. Unlike the (lazy-training) infinite-width limit, where TL is ineffective, we demonstrate that in the proportional limit TL occurs due to a renormalized source-target kernel that quantifies their relatedness and determines whether TL is beneficial for generalization. |
| title | Statistical mechanics of transfer learning in fully-connected networks in the proportional limit |
| topic | Disordered Systems and Neural Networks Statistical Mechanics |
| url | https://arxiv.org/abs/2407.07168 |