Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916846937047040 |
|---|---|
| author | Chemnitz, Dennis Engel, Maximilian |
| author_facet | Chemnitz, Dennis Engel, Maximilian |
| contents | For overparameterized optimization tasks, such as those found in modern machine learning, global minima are generally not unique. In order to understand generalization in these settings, it is vital to study to which minimum an optimization algorithm converges. The possibility of having minima that are unstable under the dynamics imposed by the optimization algorithm limits the potential minima that the algorithm can find. In this paper, we characterize the global minima that are dynamically stable/unstable for both deterministic and stochastic gradient descent (SGD). In particular, we introduce a characteristic Lyapunov exponent that depends on the local dynamics around a global minimum and rigorously prove that the sign of this Lyapunov exponent determines whether SGD can accumulate at the respective global minimum. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_20209 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learning Chemnitz, Dennis Engel, Maximilian Machine Learning Dynamical Systems Probability For overparameterized optimization tasks, such as those found in modern machine learning, global minima are generally not unique. In order to understand generalization in these settings, it is vital to study to which minimum an optimization algorithm converges. The possibility of having minima that are unstable under the dynamics imposed by the optimization algorithm limits the potential minima that the algorithm can find. In this paper, we characterize the global minima that are dynamically stable/unstable for both deterministic and stochastic gradient descent (SGD). In particular, we introduce a characteristic Lyapunov exponent that depends on the local dynamics around a global minimum and rigorously prove that the sign of this Lyapunov exponent determines whether SGD can accumulate at the respective global minimum. |
| title | Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learning |
| topic | Machine Learning Dynamical Systems Probability |
| url | https://arxiv.org/abs/2407.20209 |