Proportional infinite-width infinite-depth limit for deep linear neural networks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929601922465792 |
|---|---|
| author | Bassetti, Federico Ladelli, Lucia Rotondo, Pietro |
| author_facet | Bassetti, Federico Ladelli, Lucia Rotondo, Pietro |
| contents | We study the distributional properties of linear neural networks with random parameters in the context of large networks, where the number of layers diverges in proportion to the number of neurons per layer. Prior works have shown that in the infinite-width regime, where the number of neurons per layer grows to infinity while the depth remains fixed, neural networks converge to a Gaussian process, known as the Neural Network Gaussian Process. However, this Gaussian limit sacrifices descriptive power, as it lacks the ability to learn dependent features and produce output correlations that reflect observed labels. Motivated by these limitations, we explore the joint proportional limit in which both depth and width diverge but maintain a constant ratio, yielding a non-Gaussian distribution that retains correlations between outputs. Our contribution extends previous works by rigorously characterizing, for linear activation functions, the limiting distribution as a nontrivial mixture of Gaussians. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_15267 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Proportional infinite-width infinite-depth limit for deep linear neural networks Bassetti, Federico Ladelli, Lucia Rotondo, Pietro Machine Learning Disordered Systems and Neural Networks Probability 60F05, 60H05, 62E2 We study the distributional properties of linear neural networks with random parameters in the context of large networks, where the number of layers diverges in proportion to the number of neurons per layer. Prior works have shown that in the infinite-width regime, where the number of neurons per layer grows to infinity while the depth remains fixed, neural networks converge to a Gaussian process, known as the Neural Network Gaussian Process. However, this Gaussian limit sacrifices descriptive power, as it lacks the ability to learn dependent features and produce output correlations that reflect observed labels. Motivated by these limitations, we explore the joint proportional limit in which both depth and width diverge but maintain a constant ratio, yielding a non-Gaussian distribution that retains correlations between outputs. Our contribution extends previous works by rigorously characterizing, for linear activation functions, the limiting distribution as a nontrivial mixture of Gaussians. |
| title | Proportional infinite-width infinite-depth limit for deep linear neural networks |
| topic | Machine Learning Disordered Systems and Neural Networks Probability 60F05, 60H05, 62E2 |
| url | https://arxiv.org/abs/2411.15267 |