Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910999653646336 |
|---|---|
| author | Jones-McCormick, Taj Jagannath, Aukosh Sen, Subhabrata |
| author_facet | Jones-McCormick, Taj Jagannath, Aukosh Sen, Subhabrata |
| contents | Unsupervised pre-training and transfer learning are commonly used techniques to initialize training algorithms for neural networks, particularly in settings with limited labeled data. In this paper, we study the effects of unsupervised pre-training and transfer learning on the sample complexity of high-dimensional supervised learning. Specifically, we consider the problem of training a single-layer neural network via online stochastic gradient descent. We establish that pre-training and transfer learning (under concept shift) reduce sample complexity by polynomial factors (in the dimension) under very general assumptions. We also uncover some surprising settings where pre-training grants exponential improvement over random initialization in terms of sample complexity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16849 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models Jones-McCormick, Taj Jagannath, Aukosh Sen, Subhabrata Machine Learning Unsupervised pre-training and transfer learning are commonly used techniques to initialize training algorithms for neural networks, particularly in settings with limited labeled data. In this paper, we study the effects of unsupervised pre-training and transfer learning on the sample complexity of high-dimensional supervised learning. Specifically, we consider the problem of training a single-layer neural network via online stochastic gradient descent. We establish that pre-training and transfer learning (under concept shift) reduce sample complexity by polynomial factors (in the dimension) under very general assumptions. We also uncover some surprising settings where pre-training grants exponential improvement over random initialization in terms of sample complexity. |
| title | Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.16849 |