Random Matrix Theory for Stochastic Gradient Descent
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912172061229056 |
|---|---|
| author | Park, Chanju Favoni, Matteo Lucini, Biagio Aarts, Gert |
| author_facet | Park, Chanju Favoni, Matteo Lucini, Biagio Aarts, Gert |
| contents | Investigating the dynamics of learning in machine learning algorithms is of paramount importance for understanding how and why an approach may be successful. The tools of physics and statistics provide a robust setting for such investigations. Here we apply concepts from random matrix theory to describe stochastic weight matrix dynamics, using the framework of Dyson Brownian motion. We derive the linear scaling rule between the learning rate (step size) and the batch size, and identify universal and non-universal aspects of weight matrix dynamics. We test our findings in the (near-)solvable case of the Gaussian Restricted Boltzmann Machine and in a linear one-hidden-layer neural network. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_20496 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Random Matrix Theory for Stochastic Gradient Descent Park, Chanju Favoni, Matteo Lucini, Biagio Aarts, Gert High Energy Physics - Lattice Disordered Systems and Neural Networks Machine Learning Investigating the dynamics of learning in machine learning algorithms is of paramount importance for understanding how and why an approach may be successful. The tools of physics and statistics provide a robust setting for such investigations. Here we apply concepts from random matrix theory to describe stochastic weight matrix dynamics, using the framework of Dyson Brownian motion. We derive the linear scaling rule between the learning rate (step size) and the batch size, and identify universal and non-universal aspects of weight matrix dynamics. We test our findings in the (near-)solvable case of the Gaussian Restricted Boltzmann Machine and in a linear one-hidden-layer neural network. |
| title | Random Matrix Theory for Stochastic Gradient Descent |
| topic | High Energy Physics - Lattice Disordered Systems and Neural Networks Machine Learning |
| url | https://arxiv.org/abs/2412.20496 |