Fully stochastic trust-region methods with Barzilai-Borwein steplengths
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912511281856512 |
|---|---|
| author | Bellavia, Stefania Morini, Benedetta Yousefi, Mahsa |
| author_facet | Bellavia, Stefania Morini, Benedetta Yousefi, Mahsa |
| contents | We investigate stochastic gradient methods and stochastic counterparts of the Barzilai-Borwein steplengths and their application to finite-sum minimization problems. Our proposal is based on the Trust-Region-ish (TRish) framework introduced in [F. E. Curtis, K. Scheinberg, R. Shi, {\it A stochastic trust region algorithm based on careful step normalization}, Informs Journal on Optimization, 1, 2019]. The new framework, named TRishBB, aims to enhance the performance of TRish and at reducing the computational cost of the second-order TRish variant. We propose three different methods belonging to the TRishBB framework and present the convergence analysis for possibly nonconvex objective functions, considering biased and unbiased gradient approximations. Our analysis requires neither diminishing step-sizes nor full gradient evaluation. The numerical experiments in machine learning applications demonstrate the effectiveness of applying the Barzilai-Borwein steplength with stochastic gradients and show improved testing accuracy compared to the TRish method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_12180 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fully stochastic trust-region methods with Barzilai-Borwein steplengths Bellavia, Stefania Morini, Benedetta Yousefi, Mahsa Optimization and Control Numerical Analysis We investigate stochastic gradient methods and stochastic counterparts of the Barzilai-Borwein steplengths and their application to finite-sum minimization problems. Our proposal is based on the Trust-Region-ish (TRish) framework introduced in [F. E. Curtis, K. Scheinberg, R. Shi, {\it A stochastic trust region algorithm based on careful step normalization}, Informs Journal on Optimization, 1, 2019]. The new framework, named TRishBB, aims to enhance the performance of TRish and at reducing the computational cost of the second-order TRish variant. We propose three different methods belonging to the TRishBB framework and present the convergence analysis for possibly nonconvex objective functions, considering biased and unbiased gradient approximations. Our analysis requires neither diminishing step-sizes nor full gradient evaluation. The numerical experiments in machine learning applications demonstrate the effectiveness of applying the Barzilai-Borwein steplength with stochastic gradients and show improved testing accuracy compared to the TRish method. |
| title | Fully stochastic trust-region methods with Barzilai-Borwein steplengths |
| topic | Optimization and Control Numerical Analysis |
| url | https://arxiv.org/abs/2412.12180 |