Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909138636767232 |
|---|---|
| author | Lin, Wu Duruisseaux, Valentin Leok, Melvin Nielsen, Frank Khan, Mohammad Emtiyaz Schmidt, Mark |
| author_facet | Lin, Wu Duruisseaux, Valentin Leok, Melvin Nielsen, Frank Khan, Mohammad Emtiyaz Schmidt, Mark |
| contents | Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations. Here, we simplify such difficulties for a class of sparse or structured symmetric positive-definite matrices with the affine-invariant metric. We do so by proposing a generalized version of the Riemannian normal coordinates that dynamically orthonormalizes the metric and locally converts the problem into an unconstrained problem in the Euclidean space. We use our approach to simplify existing approaches for structured covariances and develop matrix-inverse-free $2^\text{nd}$-order optimizers for deep learning with low precision by using only matrix multiplications. Code: https://github.com/yorkerlin/StructuredNGD-DL |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_09738 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning Lin, Wu Duruisseaux, Valentin Leok, Melvin Nielsen, Frank Khan, Mohammad Emtiyaz Schmidt, Mark Machine Learning Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations. Here, we simplify such difficulties for a class of sparse or structured symmetric positive-definite matrices with the affine-invariant metric. We do so by proposing a generalized version of the Riemannian normal coordinates that dynamically orthonormalizes the metric and locally converts the problem into an unconstrained problem in the Euclidean space. We use our approach to simplify existing approaches for structured covariances and develop matrix-inverse-free $2^\text{nd}$-order optimizers for deep learning with low precision by using only matrix multiplications. Code: https://github.com/yorkerlin/StructuredNGD-DL |
| title | Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2302.09738 |