Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916332286509056 |
|---|---|
| author | Lin, Wu Dangel, Felix Eschenhagen, Runa Neklyudov, Kirill Kristiadi, Agustinus Turner, Richard E. Makhzani, Alireza |
| author_facet | Lin, Wu Dangel, Felix Eschenhagen, Runa Neklyudov, Kirill Kristiadi, Agustinus Turner, Richard E. Makhzani, Alireza |
| contents | Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their preconditioning Kronecker factors are dense, and numerically unstable in low precision as they require matrix inversion or decomposition. These limitations render such methods unpopular for modern mixed-precision training. We address them by (i) formulating an inverse-free KFAC update and (ii) imposing structures in the Kronecker factors, resulting in structured inverse-free natural gradient descent (SINGD). On modern neural networks, we show that SINGD is memory-efficient and numerically robust, in contrast to KFAC, and often outperforms AdamW even in half precision. Our work closes a gap between first- and second-order methods in modern low-precision training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_05705 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC Lin, Wu Dangel, Felix Eschenhagen, Runa Neklyudov, Kirill Kristiadi, Agustinus Turner, Richard E. Makhzani, Alireza Machine Learning Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their preconditioning Kronecker factors are dense, and numerically unstable in low precision as they require matrix inversion or decomposition. These limitations render such methods unpopular for modern mixed-precision training. We address them by (i) formulating an inverse-free KFAC update and (ii) imposing structures in the Kronecker factors, resulting in structured inverse-free natural gradient descent (SINGD). On modern neural networks, we show that SINGD is memory-efficient and numerically robust, in contrast to KFAC, and often outperforms AdamW even in half precision. Our work closes a gap between first- and second-order methods in modern low-precision training. |
| title | Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2312.05705 |