$μ^2$-SGD: Stable Stochastic Optimization via a Double Momentum Mechanism
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916643107504128 |
|---|---|
| author | Dahan, Tehila Levy, Kfir Y. |
| author_facet | Dahan, Tehila Levy, Kfir Y. |
| contents | We consider stochastic convex optimization problems where the objective is an expectation over smooth functions. For this setting we suggest a novel gradient estimate that combines two recent mechanism that are related to notion of momentum. Then, we design an SGD-style algorithm as well as an accelerated version that make use of this new estimator, and demonstrate the robustness of these new approaches to the choice of the learning rate. Concretely, we show that these approaches obtain the optimal convergence rates for both noiseless and noisy case with the same choice of fixed learning rate. Moreover, for the noisy case we show that these approaches achieve the same optimal bound for a very wide range of learning rates. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_04172 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | $μ^2$-SGD: Stable Stochastic Optimization via a Double Momentum Mechanism Dahan, Tehila Levy, Kfir Y. Machine Learning Optimization and Control We consider stochastic convex optimization problems where the objective is an expectation over smooth functions. For this setting we suggest a novel gradient estimate that combines two recent mechanism that are related to notion of momentum. Then, we design an SGD-style algorithm as well as an accelerated version that make use of this new estimator, and demonstrate the robustness of these new approaches to the choice of the learning rate. Concretely, we show that these approaches obtain the optimal convergence rates for both noiseless and noisy case with the same choice of fixed learning rate. Moreover, for the noisy case we show that these approaches achieve the same optimal bound for a very wide range of learning rates. |
| title | $μ^2$-SGD: Stable Stochastic Optimization via a Double Momentum Mechanism |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2304.04172 |