Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916968463859712 |
|---|---|
| author | Lee, Youngkyu Kopaničáková, Alena Karniadakis, George Em |
| author_facet | Lee, Youngkyu Kopaničáková, Alena Karniadakis, George Em |
| contents | We introduce a novel two-level overlapping additive Schwarz preconditioner for accelerating the training of scientific machine learning applications. The design of the proposed preconditioner is motivated by the nonlinear two-level overlapping additive Schwarz preconditioner. The neural network parameters are decomposed into groups (subdomains) with overlapping regions. In addition, the network's feed-forward structure is indirectly imposed through a novel subdomain-wise synchronization strategy and a coarse-level training step. Through a series of numerical experiments, which consider physics-informed neural networks and operator learning approaches, we demonstrate that the proposed two-level preconditioner significantly speeds up the convergence of the standard (LBFGS) optimizer while also yielding more accurate machine learning models. Moreover, the devised preconditioner is designed to take advantage of model-parallel computations, which can further reduce the training time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10997 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications Lee, Youngkyu Kopaničáková, Alena Karniadakis, George Em Numerical Analysis Machine Learning Optimization and Control 90C30, 90C26, 90C06, 65M55, 68T07 We introduce a novel two-level overlapping additive Schwarz preconditioner for accelerating the training of scientific machine learning applications. The design of the proposed preconditioner is motivated by the nonlinear two-level overlapping additive Schwarz preconditioner. The neural network parameters are decomposed into groups (subdomains) with overlapping regions. In addition, the network's feed-forward structure is indirectly imposed through a novel subdomain-wise synchronization strategy and a coarse-level training step. Through a series of numerical experiments, which consider physics-informed neural networks and operator learning approaches, we demonstrate that the proposed two-level preconditioner significantly speeds up the convergence of the standard (LBFGS) optimizer while also yielding more accurate machine learning models. Moreover, the devised preconditioner is designed to take advantage of model-parallel computations, which can further reduce the training time. |
| title | Two-level overlapping additive Schwarz preconditioner for training scientific machine learning applications |
| topic | Numerical Analysis Machine Learning Optimization and Control 90C30, 90C26, 90C06, 65M55, 68T07 |
| url | https://arxiv.org/abs/2406.10997 |