Gradient descent provably escapes saddle points in the training of shallow ReLU networks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2022
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917772457410560 |
|---|---|
| author | Cheridito, Patrick Jentzen, Arnulf Rossmannek, Florian |
| author_facet | Cheridito, Patrick Jentzen, Arnulf Rossmannek, Florian |
| contents | Dynamical systems theory has recently been applied in optimization to prove that gradient descent algorithms bypass so-called strict saddle points of the loss function. However, in many modern machine learning applications, the required regularity conditions are not satisfied. In this paper, we prove a variant of the relevant dynamical systems result, a center-stable manifold theorem, in which we relax some of the regularity requirements. We explore its relevance for various machine learning tasks, with a particular focus on shallow rectified linear unit (ReLU) and leaky ReLU networks with scalar input. Building on a detailed examination of critical points of the square integral loss function for shallow ReLU and leaky ReLU networks relative to an affine target function, we show that gradient descent circumvents most saddle points. Furthermore, we prove convergence to global minima under favourable initialization conditions, quantified by an explicit threshold on the limiting loss. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2208_02083 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Gradient descent provably escapes saddle points in the training of shallow ReLU networks Cheridito, Patrick Jentzen, Arnulf Rossmannek, Florian Machine Learning Dynamical Systems Optimization and Control 68T07, 37D10 I.2.6; G.1.6 Dynamical systems theory has recently been applied in optimization to prove that gradient descent algorithms bypass so-called strict saddle points of the loss function. However, in many modern machine learning applications, the required regularity conditions are not satisfied. In this paper, we prove a variant of the relevant dynamical systems result, a center-stable manifold theorem, in which we relax some of the regularity requirements. We explore its relevance for various machine learning tasks, with a particular focus on shallow rectified linear unit (ReLU) and leaky ReLU networks with scalar input. Building on a detailed examination of critical points of the square integral loss function for shallow ReLU and leaky ReLU networks relative to an affine target function, we show that gradient descent circumvents most saddle points. Furthermore, we prove convergence to global minima under favourable initialization conditions, quantified by an explicit threshold on the limiting loss. |
| title | Gradient descent provably escapes saddle points in the training of shallow ReLU networks |
| topic | Machine Learning Dynamical Systems Optimization and Control 68T07, 37D10 I.2.6; G.1.6 |
| url | https://arxiv.org/abs/2208.02083 |