Spectral-factorized Positive-definite Curvature Learning for NN Training
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908288570884096 |
|---|---|
| author | Lin, Wu Dangel, Felix Eschenhagen, Runa Bae, Juhan Turner, Richard E. Grosse, Roger B. |
| author_facet | Lin, Wu Dangel, Felix Eschenhagen, Runa Bae, Juhan Turner, Richard E. Grosse, Roger B. |
| contents | Many training methods, such as Adam(W) and Shampoo, learn a positive-definite curvature matrix and apply an inverse root before preconditioning. Recently, non-diagonal training methods, such as Shampoo, have gained significant attention; however, they remain computationally inefficient and are limited to specific types of curvature information due to the costly matrix root computation via matrix decomposition. To address this, we propose a Riemannian optimization approach that dynamically adapts spectral-factorized positive-definite curvature estimates, enabling the efficient application of arbitrary matrix roots and generic curvature learning. We demonstrate the efficacy and versatility of our approach in positive-definite matrix optimization and covariance adaptation for gradient-free optimization, as well as its efficiency in curvature learning for neural net training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06268 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Spectral-factorized Positive-definite Curvature Learning for NN Training Lin, Wu Dangel, Felix Eschenhagen, Runa Bae, Juhan Turner, Richard E. Grosse, Roger B. Machine Learning Many training methods, such as Adam(W) and Shampoo, learn a positive-definite curvature matrix and apply an inverse root before preconditioning. Recently, non-diagonal training methods, such as Shampoo, have gained significant attention; however, they remain computationally inefficient and are limited to specific types of curvature information due to the costly matrix root computation via matrix decomposition. To address this, we propose a Riemannian optimization approach that dynamically adapts spectral-factorized positive-definite curvature estimates, enabling the efficient application of arbitrary matrix roots and generic curvature learning. We demonstrate the efficacy and versatility of our approach in positive-definite matrix optimization and covariance adaptation for gradient-free optimization, as well as its efficiency in curvature learning for neural net training. |
| title | Spectral-factorized Positive-definite Curvature Learning for NN Training |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2502.06268 |