DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915607355588608 |
|---|---|
| author | Yudin, Nikolay Grishina, Ekaterina Veprikov, Andrey Beznosikov, Alexandr Rakhuba, Maxim |
| author_facet | Yudin, Nikolay Grishina, Ekaterina Veprikov, Andrey Beznosikov, Alexandr Rakhuba, Maxim |
| contents | Recently, optimizers that explicitly treat weights as matrices, rather than flattened vectors, have demonstrated their effectiveness. This perspective naturally leads to structured approximations of the Fisher matrix as preconditioners, where the matrix view induces a Kronecker-factorized form that enables memory-efficient representation. However, constructing such approximations both efficiently and accurately remains an open challenge, since obtaining the optimal factorization is resource-intensive and practical methods therefore rely on heuristic design choices. In this work, we introduce a novel approach that leverages projector-splitting integrators to construct effective preconditioners. Our optimizer, DyKAF (Dynamical Kronecker Approximation of the Fisher Matrix), consistently improves the Fisher matrix approximation quality. Experiments on large language model pre-training and fine-tuning demonstrate that DyKAF outperforms existing optimizers across a range of evaluation metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_06477 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning Yudin, Nikolay Grishina, Ekaterina Veprikov, Andrey Beznosikov, Alexandr Rakhuba, Maxim Machine Learning Numerical Analysis Optimization and Control 68T07, 65F55, 15A18 Recently, optimizers that explicitly treat weights as matrices, rather than flattened vectors, have demonstrated their effectiveness. This perspective naturally leads to structured approximations of the Fisher matrix as preconditioners, where the matrix view induces a Kronecker-factorized form that enables memory-efficient representation. However, constructing such approximations both efficiently and accurately remains an open challenge, since obtaining the optimal factorization is resource-intensive and practical methods therefore rely on heuristic design choices. In this work, we introduce a novel approach that leverages projector-splitting integrators to construct effective preconditioners. Our optimizer, DyKAF (Dynamical Kronecker Approximation of the Fisher Matrix), consistently improves the Fisher matrix approximation quality. Experiments on large language model pre-training and fine-tuning demonstrate that DyKAF outperforms existing optimizers across a range of evaluation metrics. |
| title | DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning |
| topic | Machine Learning Numerical Analysis Optimization and Control 68T07, 65F55, 15A18 |
| url | https://arxiv.org/abs/2511.06477 |