Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.12709 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917256009613312 |
|---|---|
| author | Olsen, Brian Richard Fatehmanesh, Sam Xiao, Frank Kumarappan, Adarsh Gajula, Anirudh |
| author_facet | Olsen, Brian Richard Fatehmanesh, Sam Xiao, Frank Kumarappan, Adarsh Gajula, Anirudh |
| contents | Deep neural networks have revolutionized machine learning, yet their training dynamics remain theoretically unclear-we develop a continuous-time, matrix-valued stochastic differential equation (SDE) framework that rigorously connects the microscopic dynamics of SGD to the macroscopic evolution of singular-value spectra in weight matrices. We derive exact SDEs showing that squared singular values follow Dyson Brownian motion with eigenvalue repulsion, and characterize stationary distributions as gamma-type densities with power-law tails, providing the first theoretical explanation for the empirically observed 'bulk+tail' spectral structure in trained networks. Through controlled experiments on transformer and MLP architectures, we validate our theoretical predictions and demonstrate quantitative agreement between SDE-based forecasts and observed spectral evolution, providing a rigorous foundation for understanding why deep learning works. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_12709 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From SGD to Spectra: A Theory of Neural Network Weight Dynamics Olsen, Brian Richard Fatehmanesh, Sam Xiao, Frank Kumarappan, Adarsh Gajula, Anirudh Machine Learning Deep neural networks have revolutionized machine learning, yet their training dynamics remain theoretically unclear-we develop a continuous-time, matrix-valued stochastic differential equation (SDE) framework that rigorously connects the microscopic dynamics of SGD to the macroscopic evolution of singular-value spectra in weight matrices. We derive exact SDEs showing that squared singular values follow Dyson Brownian motion with eigenvalue repulsion, and characterize stationary distributions as gamma-type densities with power-law tails, providing the first theoretical explanation for the empirically observed 'bulk+tail' spectral structure in trained networks. Through controlled experiments on transformer and MLP architectures, we validate our theoretical predictions and demonstrate quantitative agreement between SDE-based forecasts and observed spectral evolution, providing a rigorous foundation for understanding why deep learning works. |
| title | From SGD to Spectra: A Theory of Neural Network Weight Dynamics |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2507.12709 |