An adaptive framework for first-order gradient methods
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918338170454016 |
|---|---|
| author | Hu, Xiaozhe Pollock, Sara Xue, Zhongqin Zhu, Yunrong |
| author_facet | Hu, Xiaozhe Pollock, Sara Xue, Zhongqin Zhu, Yunrong |
| contents | Gradient methods are widely used in optimization problems. In practice, while the smoothness parameter can be estimated utilizing techniques such as backtracking, estimating the strong convexity parameter remains a challenge; moreover, even with the optimal parameter choice, convergence can be slow. In this work, we propose a framework for dynamically adapting the step size and momentum parameters in first-order gradient methods for the optimization problem, without prior knowledge of the strong convexity parameter. The main idea is to use the geometric average of the ratios of successive residual norms as an empirical estimate of the upper bound on the convergence rate, which in turn allows us to adaptively update the algorithm parameters. The resulting algorithms are simple to implement, yet efficient in practice, requiring only a few additional computations on existing information. The proposed adaptive gradient methods are shown to converge at least as fast as gradient descent for quadratic optimization problems. Numerical experiments on both quadratic and nonlinear problems validate the effectiveness of the proposed adaptive algorithms. The results show that the adaptive algorithms are comparable to their counterparts using optimal parameters, and in some cases, they capture local information and exhibit improved performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_13620 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | An adaptive framework for first-order gradient methods Hu, Xiaozhe Pollock, Sara Xue, Zhongqin Zhu, Yunrong Optimization and Control Numerical Analysis 65K05, 65K10, 68Q25, 90C25 Gradient methods are widely used in optimization problems. In practice, while the smoothness parameter can be estimated utilizing techniques such as backtracking, estimating the strong convexity parameter remains a challenge; moreover, even with the optimal parameter choice, convergence can be slow. In this work, we propose a framework for dynamically adapting the step size and momentum parameters in first-order gradient methods for the optimization problem, without prior knowledge of the strong convexity parameter. The main idea is to use the geometric average of the ratios of successive residual norms as an empirical estimate of the upper bound on the convergence rate, which in turn allows us to adaptively update the algorithm parameters. The resulting algorithms are simple to implement, yet efficient in practice, requiring only a few additional computations on existing information. The proposed adaptive gradient methods are shown to converge at least as fast as gradient descent for quadratic optimization problems. Numerical experiments on both quadratic and nonlinear problems validate the effectiveness of the proposed adaptive algorithms. The results show that the adaptive algorithms are comparable to their counterparts using optimal parameters, and in some cases, they capture local information and exhibit improved performance. |
| title | An adaptive framework for first-order gradient methods |
| topic | Optimization and Control Numerical Analysis 65K05, 65K10, 68Q25, 90C25 |
| url | https://arxiv.org/abs/2602.13620 |