Local Linear Convergence of Infeasible Optimization with Orthogonal Constraints
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929619809075200 |
|---|---|
| author | Sun, Youbang Chen, Shixiang Garcia, Alfredo Shahrampour, Shahin |
| author_facet | Sun, Youbang Chen, Shixiang Garcia, Alfredo Shahrampour, Shahin |
| contents | Many classical and modern machine learning algorithms require solving optimization tasks under orthogonality constraints. Solving these tasks with feasible methods requires a gradient descent update followed by a retraction operation on the Stiefel manifold, which can be computationally expensive. Recently, an infeasible retraction-free approach, termed the landing algorithm, was proposed as an efficient alternative. Motivated by the common occurrence of orthogonality constraints in tasks such as principle component analysis and training of deep neural networks, this paper studies the landing algorithm and establishes a novel linear convergence rate for smooth non-convex functions using only a local Riemannian PŁ condition. Numerical experiments demonstrate that the landing algorithm performs on par with the state-of-the-art retraction-based methods with substantially reduced computational overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_05689 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Local Linear Convergence of Infeasible Optimization with Orthogonal Constraints Sun, Youbang Chen, Shixiang Garcia, Alfredo Shahrampour, Shahin Optimization and Control Machine Learning Many classical and modern machine learning algorithms require solving optimization tasks under orthogonality constraints. Solving these tasks with feasible methods requires a gradient descent update followed by a retraction operation on the Stiefel manifold, which can be computationally expensive. Recently, an infeasible retraction-free approach, termed the landing algorithm, was proposed as an efficient alternative. Motivated by the common occurrence of orthogonality constraints in tasks such as principle component analysis and training of deep neural networks, this paper studies the landing algorithm and establishes a novel linear convergence rate for smooth non-convex functions using only a local Riemannian PŁ condition. Numerical experiments demonstrate that the landing algorithm performs on par with the state-of-the-art retraction-based methods with substantially reduced computational overhead. |
| title | Local Linear Convergence of Infeasible Optimization with Orthogonal Constraints |
| topic | Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2412.05689 |