Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910021753765888 |
|---|---|
| author | DePavia, Adela Cruzado, Jose Liang, Jiayou Charisopoulos, Vasileios Willett, Rebecca |
| author_facet | DePavia, Adela Cruzado, Jose Liang, Jiayou Charisopoulos, Vasileios Willett, Rebecca |
| contents | Adaptive optimization algorithms -- such as Adagrad, Adam, and their variants -- have found widespread use in machine learning, signal processing and many other settings. Several methods in this family are not rotationally equivariant, meaning that simple reparameterizations (i.e. change of basis) can drastically affect their convergence. However, their sensitivity to the choice of parameterization has not been systematically studied; it is not clear how to identify a "favorable" change of basis in which these methods perform best. In this paper we propose a reparameterization method and demonstrate both theoretically and empirically its potential to improve their convergence behavior. Our method is an orthonormal transformation based on the expected gradient outer product (EGOP) matrix, which can be approximated using either full-batch or stochastic gradient oracles. We show that for a broad class of functions, the sensitivity of adaptive algorithms to choice-of-basis is influenced by the decay of the EGOP matrix spectrum. We illustrate the potential impact of EGOP reparameterization by presenting empirical evidence and theoretical arguments that common machine learning tasks with "natural" data exhibit EGOP spectral decay. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01594 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization DePavia, Adela Cruzado, Jose Liang, Jiayou Charisopoulos, Vasileios Willett, Rebecca Machine Learning Optimization and Control 90C06, 65K10, 90C30, 68T07 Adaptive optimization algorithms -- such as Adagrad, Adam, and their variants -- have found widespread use in machine learning, signal processing and many other settings. Several methods in this family are not rotationally equivariant, meaning that simple reparameterizations (i.e. change of basis) can drastically affect their convergence. However, their sensitivity to the choice of parameterization has not been systematically studied; it is not clear how to identify a "favorable" change of basis in which these methods perform best. In this paper we propose a reparameterization method and demonstrate both theoretically and empirically its potential to improve their convergence behavior. Our method is an orthonormal transformation based on the expected gradient outer product (EGOP) matrix, which can be approximated using either full-batch or stochastic gradient oracles. We show that for a broad class of functions, the sensitivity of adaptive algorithms to choice-of-basis is influenced by the decay of the EGOP matrix spectrum. We illustrate the potential impact of EGOP reparameterization by presenting empirical evidence and theoretical arguments that common machine learning tasks with "natural" data exhibit EGOP spectral decay. |
| title | Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization |
| topic | Machine Learning Optimization and Control 90C06, 65K10, 90C30, 68T07 |
| url | https://arxiv.org/abs/2502.01594 |