An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912427073863680 |
|---|---|
| author | Engquist, Björn Ren, Kui Yang, Yunan |
| author_facet | Engquist, Björn Ren, Kui Yang, Yunan |
| contents | We propose a new gradient descent algorithm with added stochastic terms for finding the global optimizers of nonconvex optimization problems. A key component in the algorithm is the adaptive tuning of the randomness based on the value of the objective function. In the language of simulated annealing, the temperature is state-dependent. With this, we prove the global convergence of the algorithm with an algebraic rate both in probability and in the parameter space. This is a significant improvement over the classical rate from using a more straightforward control of the noise term. The convergence proof is based on the actual discrete setup of the algorithm, not just its continuous limit as often done in the literature. We also present several numerical examples to demonstrate the efficiency and robustness of the algorithm for reasonably complex objective functions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2204_05923 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization Engquist, Björn Ren, Kui Yang, Yunan Optimization and Control Machine Learning 90C26, 90C15, 65K05 We propose a new gradient descent algorithm with added stochastic terms for finding the global optimizers of nonconvex optimization problems. A key component in the algorithm is the adaptive tuning of the randomness based on the value of the objective function. In the language of simulated annealing, the temperature is state-dependent. With this, we prove the global convergence of the algorithm with an algebraic rate both in probability and in the parameter space. This is a significant improvement over the classical rate from using a more straightforward control of the noise term. The convergence proof is based on the actual discrete setup of the algorithm, not just its continuous limit as often done in the literature. We also present several numerical examples to demonstrate the efficiency and robustness of the algorithm for reasonably complex objective functions. |
| title | An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization |
| topic | Optimization and Control Machine Learning 90C26, 90C15, 65K05 |
| url | https://arxiv.org/abs/2204.05923 |