Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917709355155456 |
|---|---|
| author | Ziyin, Liu Li, Botao Galanti, Tomer Ueda, Masahito |
| author_facet | Ziyin, Liu Li, Botao Galanti, Tomer Ueda, Masahito |
| contents | Characterizing and understanding the dynamics of stochastic gradient descent (SGD) around saddle points remains an open problem. We first show that saddle points in neural networks can be divided into two types, among which the Type-II saddles are especially difficult to escape from because the gradient noise vanishes at the saddle. The dynamics of SGD around these saddles are thus to leading order described by a random matrix product process, and it is thus natural to study the dynamics of SGD around these saddles using the notion of probabilistic stability and the related Lyapunov exponent. Theoretically, we link the study of SGD dynamics to well-known concepts in ergodic theory, which we leverage to show that saddle points can be either attractive or repulsive for SGD, and its dynamics can be classified into four different phases, depending on the signal-to-noise ratio in the gradient close to the saddle. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2303_13093 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent Ziyin, Liu Li, Botao Galanti, Tomer Ueda, Masahito Machine Learning Optimization and Control Data Analysis, Statistics and Probability Characterizing and understanding the dynamics of stochastic gradient descent (SGD) around saddle points remains an open problem. We first show that saddle points in neural networks can be divided into two types, among which the Type-II saddles are especially difficult to escape from because the gradient noise vanishes at the saddle. The dynamics of SGD around these saddles are thus to leading order described by a random matrix product process, and it is thus natural to study the dynamics of SGD around these saddles using the notion of probabilistic stability and the related Lyapunov exponent. Theoretically, we link the study of SGD dynamics to well-known concepts in ergodic theory, which we leverage to show that saddle points can be either attractive or repulsive for SGD, and its dynamics can be classified into four different phases, depending on the signal-to-noise ratio in the gradient close to the saddle. |
| title | Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent |
| topic | Machine Learning Optimization and Control Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2303.13093 |