Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.07729 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909916171599872 |
|---|---|
| author | Carlon, André Espath, Luis Tempone, Raúl |
| author_facet | Carlon, André Espath, Luis Tempone, Raúl |
| contents | Quasi-Newton methods are ubiquitous in deterministic local search due to their efficiency and low computational cost. This class of methods uses the history of gradient evaluations to approximate second-order derivatives. However, only noisy gradient observations are accessible in stochastic optimization; thus, deriving quasi-Newton methods in this setting is challenging. Although most existing quasi-Newton methods for stochastic optimization rely on deterministic equations that are modified to circumvent noise, we propose a new approach inspired by Bayesian inference to assimilate noisy gradient information and derive the stochastic counterparts to standard quasi-Newton methods. We focus on the derivations of stochastic BFGS and L-BFGS, but our methodology can also be employed to derive stochastic analogs of other quasi-Newton methods. The resulting stochastic BFGS (S-BFGS) and stochastic L-BFGS (L-S-BFGS) can effectively learn an inverse Hessian approximation even with small batch sizes. For a problem of dimension $d$, the iteration cost of S-BFGS is $\mathcal{O}(d^2)$, and the cost of L-S-BFGS is $\mathcal{O}(d)$. Numerical experiments with a dimensionality of up to $30,720$ demonstrate the efficiency and robustness of the proposed method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07729 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient Stochastic BFGS methods Inspired by Bayesian Principles Carlon, André Espath, Luis Tempone, Raúl Optimization and Control Quasi-Newton methods are ubiquitous in deterministic local search due to their efficiency and low computational cost. This class of methods uses the history of gradient evaluations to approximate second-order derivatives. However, only noisy gradient observations are accessible in stochastic optimization; thus, deriving quasi-Newton methods in this setting is challenging. Although most existing quasi-Newton methods for stochastic optimization rely on deterministic equations that are modified to circumvent noise, we propose a new approach inspired by Bayesian inference to assimilate noisy gradient information and derive the stochastic counterparts to standard quasi-Newton methods. We focus on the derivations of stochastic BFGS and L-BFGS, but our methodology can also be employed to derive stochastic analogs of other quasi-Newton methods. The resulting stochastic BFGS (S-BFGS) and stochastic L-BFGS (L-S-BFGS) can effectively learn an inverse Hessian approximation even with small batch sizes. For a problem of dimension $d$, the iteration cost of S-BFGS is $\mathcal{O}(d^2)$, and the cost of L-S-BFGS is $\mathcal{O}(d)$. Numerical experiments with a dimensionality of up to $30,720$ demonstrate the efficiency and robustness of the proposed method. |
| title | Efficient Stochastic BFGS methods Inspired by Bayesian Principles |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2507.07729 |