Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.12149 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914108528394240 |
|---|---|
| author | Guzmán-Cordero, Andrés Dangel, Felix Goldshlager, Gil Zeinhofer, Marius |
| author_facet | Guzmán-Cordero, Andrés Dangel, Felix Goldshlager, Gil Zeinhofer, Marius |
| contents | Natural gradient methods significantly accelerate the training of Physics-Informed Neural Networks (PINNs), but are often prohibitively costly. We introduce a suite of techniques to improve the accuracy and efficiency of energy natural gradient descent (ENGD) for PINNs. First, we leverage the Woodbury formula to dramatically reduce the computational complexity of ENGD. Second, we adapt the Subsampled Projected-Increment Natural Gradient Descent algorithm from the variational Monte Carlo literature to accelerate the convergence. Third, we explore the use of randomized algorithms to further reduce the computational cost in the case of large batch sizes. We find that randomization accelerates progress in the early stages of training for low-dimensional problems, and we identify key barriers to attaining acceleration in other scenarios. Our numerical experiments demonstrate that our methods outperform previous approaches, achieving the same $L^2$ error as the original ENGD up to $75\times$ faster. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12149 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization Guzmán-Cordero, Andrés Dangel, Felix Goldshlager, Gil Zeinhofer, Marius Machine Learning Natural gradient methods significantly accelerate the training of Physics-Informed Neural Networks (PINNs), but are often prohibitively costly. We introduce a suite of techniques to improve the accuracy and efficiency of energy natural gradient descent (ENGD) for PINNs. First, we leverage the Woodbury formula to dramatically reduce the computational complexity of ENGD. Second, we adapt the Subsampled Projected-Increment Natural Gradient Descent algorithm from the variational Monte Carlo literature to accelerate the convergence. Third, we explore the use of randomized algorithms to further reduce the computational cost in the case of large batch sizes. We find that randomization accelerates progress in the early stages of training for low-dimensional problems, and we identify key barriers to attaining acceleration in other scenarios. Our numerical experiments demonstrate that our methods outperform previous approaches, achieving the same $L^2$ error as the original ENGD up to $75\times$ faster. |
| title | Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.12149 |