Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.12188 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909140192854016 |
|---|---|
| author | Zampini, Stefano Zerbinati, Umberto Turkiyyah, George Keyes, David |
| author_facet | Zampini, Stefano Zerbinati, Umberto Turkiyyah, George Keyes, David |
| contents | In recent years, we have witnessed the emergence of scientific machine learning as a data-driven tool for the analysis, by means of deep-learning techniques, of data produced by computational science and engineering applications. At the core of these methods is the supervised training algorithm to learn the neural network realization, a highly non-convex optimization problem that is usually solved using stochastic gradient methods. However, distinct from deep-learning practice, scientific machine-learning training problems feature a much larger volume of smooth data and better characterizations of the empirical risk functions, which make them suited for conventional solvers for unconstrained optimization. We introduce a lightweight software framework built on top of the Portable and Extensible Toolkit for Scientific computation to bridge the gap between deep-learning software and conventional solvers for unconstrained minimization. We empirically demonstrate the superior efficacy of a trust region method based on the Gauss-Newton approximation of the Hessian in improving the generalization errors arising from regression tasks when learning surrogate models for a wide range of scientific machine-learning techniques and test cases. All the conventional second-order solvers tested, including L-BFGS and inexact Newton with line-search, compare favorably, either in terms of cost or accuracy, with the adaptive first-order methods used to validate the surrogate models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12188 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PETScML: Second-order solvers for training regression problems in Scientific Machine Learning Zampini, Stefano Zerbinati, Umberto Turkiyyah, George Keyes, David Machine Learning Mathematical Software Optimization and Control 65K10, 68T07, 65M70, 65Y05 I.2.5; D.2.m; G.4; G.1.6; J.2 In recent years, we have witnessed the emergence of scientific machine learning as a data-driven tool for the analysis, by means of deep-learning techniques, of data produced by computational science and engineering applications. At the core of these methods is the supervised training algorithm to learn the neural network realization, a highly non-convex optimization problem that is usually solved using stochastic gradient methods. However, distinct from deep-learning practice, scientific machine-learning training problems feature a much larger volume of smooth data and better characterizations of the empirical risk functions, which make them suited for conventional solvers for unconstrained optimization. We introduce a lightweight software framework built on top of the Portable and Extensible Toolkit for Scientific computation to bridge the gap between deep-learning software and conventional solvers for unconstrained minimization. We empirically demonstrate the superior efficacy of a trust region method based on the Gauss-Newton approximation of the Hessian in improving the generalization errors arising from regression tasks when learning surrogate models for a wide range of scientific machine-learning techniques and test cases. All the conventional second-order solvers tested, including L-BFGS and inexact Newton with line-search, compare favorably, either in terms of cost or accuracy, with the adaptive first-order methods used to validate the surrogate models. |
| title | PETScML: Second-order solvers for training regression problems in Scientific Machine Learning |
| topic | Machine Learning Mathematical Software Optimization and Control 65K10, 68T07, 65M70, 65Y05 I.2.5; D.2.m; G.4; G.1.6; J.2 |
| url | https://arxiv.org/abs/2403.12188 |