Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916875348213760 |
|---|---|
| author | Xu, Xianliang Du, Ting Kong, Wang Shan, Bin Li, Ye Huang, Zhongyi |
| author_facet | Xu, Xianliang Du, Ting Kong, Wang Shan, Bin Li, Ye Huang, Zhongyi |
| contents | The optimization algorithms are crucial in training physics-informed neural networks (PINNs), as unsuitable methods may lead to poor solutions. Compared to the common gradient descent (GD) algorithm, implicit gradient descent (IGD) outperforms it in handling certain multi-scale problems. In this paper, we provide convergence analysis for the IGD in training over-parameterized two-layer PINNs. We first derive the training dynamics of IGD in training two-layer PINNs. Then, over-parameterization allows us to prove that the randomly initialized IGD converges to a globally optimal solution at a linear convergence rate. Moreover, due to the distinct training dynamics of IGD compared to GD, the learning rate can be selected independently of the sample size and the least eigenvalue of the Gram matrix. Additionally, the novel approach used in our convergence analysis imposes a milder requirement on the network width. Finally, empirical results validate our theoretical findings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_02827 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks Xu, Xianliang Du, Ting Kong, Wang Shan, Bin Li, Ye Huang, Zhongyi Machine Learning Optimization and Control The optimization algorithms are crucial in training physics-informed neural networks (PINNs), as unsuitable methods may lead to poor solutions. Compared to the common gradient descent (GD) algorithm, implicit gradient descent (IGD) outperforms it in handling certain multi-scale problems. In this paper, we provide convergence analysis for the IGD in training over-parameterized two-layer PINNs. We first derive the training dynamics of IGD in training two-layer PINNs. Then, over-parameterization allows us to prove that the randomly initialized IGD converges to a globally optimal solution at a linear convergence rate. Moreover, due to the distinct training dynamics of IGD compared to GD, the learning rate can be selected independently of the sample size and the least eigenvalue of the Gram matrix. Additionally, the novel approach used in our convergence analysis imposes a milder requirement on the network width. Finally, empirical results validate our theoretical findings. |
| title | Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2407.02827 |