Curvature-Adaptive Perturbation and Subspace Descent for Robust Saddle Point Escape in High-Dimensional Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909767355596800 |
|---|---|
| author | Katende, Ronald Kasumba, Henry |
| author_facet | Katende, Ronald Kasumba, Henry |
| contents | High-dimensional non-convex optimization problems in engineering design, control, and learning are often hindered by saddle points, flat plateaus, and strongly anisotropic curvature. This paper develops a unified, curvature-adaptive framework that combines stochastic perturbations, adaptive learning rates, and randomized subspace descent to enhance escape efficiency and scalability. We show theoretically that gradient flow almost surely avoids strict saddles, with escape probability increasing exponentially in dimension. For noise-perturbed gradient descent, we derive explicit escape-time bounds that depend on local curvature and noise magnitude. Adaptive step sizes further reduce escape times by responding to local gradient variability, while randomized subspace descent preserves descent directions in low-dimensional projections and ensures global convergence with logarithmic dependence on dimension. Numerical experiments on nonlinear and constrained benchmarks validate these results, demonstrating faster escape, improved robustness to ill-conditioning, and lower total runtime compared to standard first- and second-order methods. The proposed approach offers practical tools for large-scale engineering optimization tasks where curvature, noise, and dimensionality interplay critically. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12604 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Curvature-Adaptive Perturbation and Subspace Descent for Robust Saddle Point Escape in High-Dimensional Optimization Katende, Ronald Kasumba, Henry Optimization and Control High-dimensional non-convex optimization problems in engineering design, control, and learning are often hindered by saddle points, flat plateaus, and strongly anisotropic curvature. This paper develops a unified, curvature-adaptive framework that combines stochastic perturbations, adaptive learning rates, and randomized subspace descent to enhance escape efficiency and scalability. We show theoretically that gradient flow almost surely avoids strict saddles, with escape probability increasing exponentially in dimension. For noise-perturbed gradient descent, we derive explicit escape-time bounds that depend on local curvature and noise magnitude. Adaptive step sizes further reduce escape times by responding to local gradient variability, while randomized subspace descent preserves descent directions in low-dimensional projections and ensures global convergence with logarithmic dependence on dimension. Numerical experiments on nonlinear and constrained benchmarks validate these results, demonstrating faster escape, improved robustness to ill-conditioning, and lower total runtime compared to standard first- and second-order methods. The proposed approach offers practical tools for large-scale engineering optimization tasks where curvature, noise, and dimensionality interplay critically. |
| title | Curvature-Adaptive Perturbation and Subspace Descent for Robust Saddle Point Escape in High-Dimensional Optimization |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2409.12604 |