Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916398271299584 |
|---|---|
| author | Köhne, Frederik Kreis, Leonie Schiela, Anton Herzog, Roland |
| author_facet | Köhne, Frederik Kreis, Leonie Schiela, Anton Herzog, Roland |
| contents | This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties and exhibits truly problem adaptive behavior on classical image classification tasks. Our framework is set in a general Hilbert space and thus enables the potential inclusion of a preconditioner through the choice of the inner product. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_16956 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent Köhne, Frederik Kreis, Leonie Schiela, Anton Herzog, Roland Optimization and Control Machine Learning This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties and exhibits truly problem adaptive behavior on classical image classification tasks. Our framework is set in a general Hilbert space and thus enables the potential inclusion of a preconditioner through the choice of the inner product. |
| title | Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent |
| topic | Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2311.16956 |