Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2022
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2206.05794 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909354253352960 |
|---|---|
| author | Galanti, Tomer Siegel, Zachary S. Gupte, Aparna Poggio, Tomaso |
| author_facet | Galanti, Tomer Siegel, Zachary S. Gupte, Aparna Poggio, Tomaso |
| contents | We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate that training with mini-batch SGD and weight decay induces a bias toward rank minimization in the weight matrices. Specifically, we show both theoretically and empirically that this bias becomes more pronounced with smaller batch sizes, higher learning rates, or stronger weight decay. Additionally, we predict and empirically confirm that weight decay is essential for this bias to occur. Unlike previous literature, our analysis does not rely on assumptions about the data, convergence, or optimality of the weight matrices, making it applicable to a wide range of neural network architectures of any width or depth. Finally, we empirically explore the connection between this bias and generalization, finding that it has a marginal effect on the test performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2206_05794 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network Galanti, Tomer Siegel, Zachary S. Gupte, Aparna Poggio, Tomaso Machine Learning We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate that training with mini-batch SGD and weight decay induces a bias toward rank minimization in the weight matrices. Specifically, we show both theoretically and empirically that this bias becomes more pronounced with smaller batch sizes, higher learning rates, or stronger weight decay. Additionally, we predict and empirically confirm that weight decay is essential for this bias to occur. Unlike previous literature, our analysis does not rely on assumptions about the data, convergence, or optimality of the weight matrices, making it applicable to a wide range of neural network architectures of any width or depth. Finally, we empirically explore the connection between this bias and generalization, finding that it has a marginal effect on the test performance. |
| title | SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2206.05794 |