Position: Curvature Matrices Should Be Democratized via Linear Operators
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909471252414464 |
|---|---|
| author | Dangel, Felix Eschenhagen, Runa Ormaniec, Weronika Fernandez, Andres Tatzel, Lukas Kristiadi, Agustinus |
| author_facet | Dangel, Felix Eschenhagen, Runa Ormaniec, Weronika Fernandez, Andres Tatzel, Lukas Kristiadi, Agustinus |
| contents | Structured large matrices are prevalent in machine learning. A particularly important class is curvature matrices like the Hessian, which are central to understanding the loss landscape of neural nets (NNs), and enable second-order optimization, uncertainty quantification, model pruning, data attribution, and more. However, curvature computations can be challenging due to the complexity of automatic differentiation, and the variety and structural assumptions of curvature proxies, like sparsity and Kronecker factorization. In this position paper, we argue that linear operators -- an interface for performing matrix-vector products -- provide a general, scalable, and user-friendly abstraction to handle curvature matrices. To support this position, we developed $\textit{curvlinops}$, a library that provides curvature matrices through a unified linear operator interface. We demonstrate with $\textit{curvlinops}$ how this interface can hide complexity, simplify applications, be extensible and interoperable with other libraries, and scale to large NNs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_19183 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Position: Curvature Matrices Should Be Democratized via Linear Operators Dangel, Felix Eschenhagen, Runa Ormaniec, Weronika Fernandez, Andres Tatzel, Lukas Kristiadi, Agustinus Machine Learning Structured large matrices are prevalent in machine learning. A particularly important class is curvature matrices like the Hessian, which are central to understanding the loss landscape of neural nets (NNs), and enable second-order optimization, uncertainty quantification, model pruning, data attribution, and more. However, curvature computations can be challenging due to the complexity of automatic differentiation, and the variety and structural assumptions of curvature proxies, like sparsity and Kronecker factorization. In this position paper, we argue that linear operators -- an interface for performing matrix-vector products -- provide a general, scalable, and user-friendly abstraction to handle curvature matrices. To support this position, we developed $\textit{curvlinops}$, a library that provides curvature matrices through a unified linear operator interface. We demonstrate with $\textit{curvlinops}$ how this interface can hide complexity, simplify applications, be extensible and interoperable with other libraries, and scale to large NNs. |
| title | Position: Curvature Matrices Should Be Democratized via Linear Operators |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2501.19183 |