Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912314827997184 |
|---|---|
| author | Tucat, Matteo Mukherjee, Anirbit Sen, Procheta Sun, Mingfei Rivasplata, Omar |
| author_facet | Tucat, Matteo Mukherjee, Anirbit Sen, Procheta Sun, Mingfei Rivasplata, Omar |
| contents | We present and analyze a novel regularized form of the gradient clipping algorithm, proving that it converges to global minima of the loss surface of deep neural networks under the squared loss, provided that the layers are of sufficient width. The algorithm presented here, dubbed $δ-$GClip, introduces a modification to gradient clipping that leads to a first-of-its-kind example of a step size scheduling for gradient descent that provably minimizes training losses of deep neural nets. We also present empirical evidence that our theoretically founded $δ-$GClip algorithm is competitive with the state-of-the-art deep learning heuristics on various neural architectures including modern transformer based architectures. The modification we do to standard gradient clipping is designed to leverage the PL* condition, a variant of the Polyak-Lojasiewicz inequality which was recently proven to be true for sufficiently wide neural networks at any depth within a neighbourhood of the initialization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_08624 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks Tucat, Matteo Mukherjee, Anirbit Sen, Procheta Sun, Mingfei Rivasplata, Omar Machine Learning Optimization and Control We present and analyze a novel regularized form of the gradient clipping algorithm, proving that it converges to global minima of the loss surface of deep neural networks under the squared loss, provided that the layers are of sufficient width. The algorithm presented here, dubbed $δ-$GClip, introduces a modification to gradient clipping that leads to a first-of-its-kind example of a step size scheduling for gradient descent that provably minimizes training losses of deep neural nets. We also present empirical evidence that our theoretically founded $δ-$GClip algorithm is competitive with the state-of-the-art deep learning heuristics on various neural architectures including modern transformer based architectures. The modification we do to standard gradient clipping is designed to leverage the PL* condition, a variant of the Polyak-Lojasiewicz inequality which was recently proven to be true for sufficiently wide neural networks at any depth within a neighbourhood of the initialization. |
| title | Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2404.08624 |