A Discrete Variational Derivation of Accelerated Methods in Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2021
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913315867852800 |
|---|---|
| author | Campos, Cédric M. Mahillo, Alejandro de Diego, David Martín |
| author_facet | Campos, Cédric M. Mahillo, Alejandro de Diego, David Martín |
| contents | Many of the new developments in machine learning are connected with gradient-based optimization methods. Recently, these methods have been studied using a variational perspective. This has opened up the possibility of introducing variational and symplectic methods using geometric integration. In particular, in this paper, we introduce variational integrators which allow us to derive different methods for optimization. Using both, Hamilton's and Lagrange-d'Alembert's principle, we derive two families of respective optimization methods in one-to-one correspondence that generalize Polyak's heavy ball and the well known Nesterov accelerated gradient method, the second of which mimics the behavior of the first reducing the oscillations of classical momentum methods. However, since the systems considered are explicitly time-dependent, the preservation of symplecticity of autonomous systems occurs here solely on the fibers. Several experiments exemplify the result. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2106_02700 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | A Discrete Variational Derivation of Accelerated Methods in Optimization Campos, Cédric M. Mahillo, Alejandro de Diego, David Martín Optimization and Control Machine Learning Numerical Analysis Differential Geometry Many of the new developments in machine learning are connected with gradient-based optimization methods. Recently, these methods have been studied using a variational perspective. This has opened up the possibility of introducing variational and symplectic methods using geometric integration. In particular, in this paper, we introduce variational integrators which allow us to derive different methods for optimization. Using both, Hamilton's and Lagrange-d'Alembert's principle, we derive two families of respective optimization methods in one-to-one correspondence that generalize Polyak's heavy ball and the well known Nesterov accelerated gradient method, the second of which mimics the behavior of the first reducing the oscillations of classical momentum methods. However, since the systems considered are explicitly time-dependent, the preservation of symplecticity of autonomous systems occurs here solely on the fibers. Several experiments exemplify the result. |
| title | A Discrete Variational Derivation of Accelerated Methods in Optimization |
| topic | Optimization and Control Machine Learning Numerical Analysis Differential Geometry |
| url | https://arxiv.org/abs/2106.02700 |