Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915057110089728 |
|---|---|
| author | Liu, Weihua Boumaraf, Said Li, Jianwu Lin, Chaochao Liu, Xiabi Niu, Lijuan Werghi, Naoufel |
| author_facet | Liu, Weihua Boumaraf, Said Li, Jianwu Lin, Chaochao Liu, Xiabi Niu, Lijuan Werghi, Naoufel |
| contents | Natural gradient descent (NGD) is a powerful optimization technique for machine learning, but the computational complexity of the inverse Fisher information matrix limits its application in training deep neural networks. To overcome this challenge, we propose a novel optimization method for training deep neural networks called structured natural gradient descent (SNGD). Theoretically, we demonstrate that optimizing the original network using NGD is equivalent to using fast gradient descent (GD) to optimize the reconstructed network with a structural transformation of the parameter matrix. Thereby, we decompose the calculation of the global Fisher information matrix into the efficient computation of local Fisher matrices via constructing local Fisher layers in the reconstructed network to speed up the training. Experimental results on various deep networks and datasets demonstrate that SNGD achieves faster convergence speed than NGD while retaining comparable solutions. Furthermore, our method outperforms traditional GDs in terms of efficiency and effectiveness. Thus, our proposed method has the potential to significantly improve the scalability and efficiency of NGD in deep learning applications. Our source code is available at https://github.com/Chaochao-Lin/SNGD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07441 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent Liu, Weihua Boumaraf, Said Li, Jianwu Lin, Chaochao Liu, Xiabi Niu, Lijuan Werghi, Naoufel Machine Learning Artificial Intelligence Natural gradient descent (NGD) is a powerful optimization technique for machine learning, but the computational complexity of the inverse Fisher information matrix limits its application in training deep neural networks. To overcome this challenge, we propose a novel optimization method for training deep neural networks called structured natural gradient descent (SNGD). Theoretically, we demonstrate that optimizing the original network using NGD is equivalent to using fast gradient descent (GD) to optimize the reconstructed network with a structural transformation of the parameter matrix. Thereby, we decompose the calculation of the global Fisher information matrix into the efficient computation of local Fisher matrices via constructing local Fisher layers in the reconstructed network to speed up the training. Experimental results on various deep networks and datasets demonstrate that SNGD achieves faster convergence speed than NGD while retaining comparable solutions. Furthermore, our method outperforms traditional GDs in terms of efficiency and effectiveness. Thus, our proposed method has the potential to significantly improve the scalability and efficiency of NGD in deep learning applications. Our source code is available at https://github.com/Chaochao-Lin/SNGD. |
| title | Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2412.07441 |