SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917261080526848 |
|---|---|
| author | Chang, Zhipeng Hao, Wenrui Liu, Nian |
| author_facet | Chang, Zhipeng Hao, Wenrui Liu, Nian |
| contents | This paper introduces a novel optimization algorithm designed for nonlinear least-squares problems. The method is derived by preconditioning the gradient descent direction using the Singular Value Decomposition (SVD) of the Jacobian. This SVD-based preconditioner is then integrated with the first- and second-moment adaptive learning rate mechanism of the Adam optimizer. We establish the local linear convergence of the proposed method under standard regularity assumptions and prove global convergence for a modified version of the algorithm under suitable conditions. The effectiveness of the approach is demonstrated experimentally across a range of tasks, including function approximation, partial differential equation (PDE) solving, and image classification on the CIFAR-10 dataset. Results show that the proposed method consistently outperforms standard Adam, achieving faster convergence and lower error in both regression and classification settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_09057 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems Chang, Zhipeng Hao, Wenrui Liu, Nian Numerical Analysis Machine Learning Optimization and Control 65K10, 90C30, 65F08, 68T07 This paper introduces a novel optimization algorithm designed for nonlinear least-squares problems. The method is derived by preconditioning the gradient descent direction using the Singular Value Decomposition (SVD) of the Jacobian. This SVD-based preconditioner is then integrated with the first- and second-moment adaptive learning rate mechanism of the Adam optimizer. We establish the local linear convergence of the proposed method under standard regularity assumptions and prove global convergence for a modified version of the algorithm under suitable conditions. The effectiveness of the approach is demonstrated experimentally across a range of tasks, including function approximation, partial differential equation (PDE) solving, and image classification on the CIFAR-10 dataset. Results show that the proposed method consistently outperforms standard Adam, achieving faster convergence and lower error in both regression and classification settings. |
| title | SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems |
| topic | Numerical Analysis Machine Learning Optimization and Control 65K10, 90C30, 65F08, 68T07 |
| url | https://arxiv.org/abs/2602.09057 |