Gaussian Universality in Neural Network Dynamics with Generalized Structured Input Distributions
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918219012374528 |
|---|---|
| author | Bae, Jaeyong Jeong, Hawoong |
| author_facet | Bae, Jaeyong Jeong, Hawoong |
| contents | Analyzing neural network dynamics via stochastic gradient descent (SGD) is crucial to building theoretical foundations for deep learning. Previous work has analyzed structured inputs within the \textit{hidden manifold model}, often under the simplifying assumption of a Gaussian distribution. We extend this framework by modeling inputs as Gaussian mixtures to better represent complex, real-world data. Through empirical and theoretical investigation, we demonstrate that with proper standardization, the learning dynamics converges to the behavior seen in the simple Gaussian case. This finding exhibits a form of universality, where diverse structured distributions yield results consistent with Gaussian assumptions, thereby strengthening the theoretical understanding of deep learning models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_00642 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Gaussian Universality in Neural Network Dynamics with Generalized Structured Input Distributions Bae, Jaeyong Jeong, Hawoong Machine Learning Disordered Systems and Neural Networks Statistical Mechanics Analyzing neural network dynamics via stochastic gradient descent (SGD) is crucial to building theoretical foundations for deep learning. Previous work has analyzed structured inputs within the \textit{hidden manifold model}, often under the simplifying assumption of a Gaussian distribution. We extend this framework by modeling inputs as Gaussian mixtures to better represent complex, real-world data. Through empirical and theoretical investigation, we demonstrate that with proper standardization, the learning dynamics converges to the behavior seen in the simple Gaussian case. This finding exhibits a form of universality, where diverse structured distributions yield results consistent with Gaussian assumptions, thereby strengthening the theoretical understanding of deep learning models. |
| title | Gaussian Universality in Neural Network Dynamics with Generalized Structured Input Distributions |
| topic | Machine Learning Disordered Systems and Neural Networks Statistical Mechanics |
| url | https://arxiv.org/abs/2405.00642 |