Efficiently Escaping Saddle Points under Generalized Smoothness via Self-Bounding Regularity
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913913282494464 |
|---|---|
| author | Cao, Daniel Yiming Chen, August Y. Sridharan, Karthik Tang, Benjamin |
| author_facet | Cao, Daniel Yiming Chen, August Y. Sridharan, Karthik Tang, Benjamin |
| contents | We study the optimization of non-convex functions that are not necessarily smooth (gradient and/or Hessian are Lipschitz) using first order methods. Smoothness is a restrictive assumption in machine learning in both theory and practice, motivating significant recent work on finding first order stationary points of functions satisfying generalizations of smoothness with first order methods. We develop a novel framework that lets us systematically study the convergence of a large class of first-order optimization algorithms (which we call decrease procedures) under generalizations of smoothness. We instantiate our framework to analyze the convergence of first order optimization algorithms to first and \textit{second} order stationary points under generalizations of smoothness. As a consequence, we establish the first convergence guarantees for first order methods to second order stationary points under generalizations of smoothness. We demonstrate that several canonical examples fall under our framework, and highlight practical implications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_04712 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficiently Escaping Saddle Points under Generalized Smoothness via Self-Bounding Regularity Cao, Daniel Yiming Chen, August Y. Sridharan, Karthik Tang, Benjamin Optimization and Control Machine Learning We study the optimization of non-convex functions that are not necessarily smooth (gradient and/or Hessian are Lipschitz) using first order methods. Smoothness is a restrictive assumption in machine learning in both theory and practice, motivating significant recent work on finding first order stationary points of functions satisfying generalizations of smoothness with first order methods. We develop a novel framework that lets us systematically study the convergence of a large class of first-order optimization algorithms (which we call decrease procedures) under generalizations of smoothness. We instantiate our framework to analyze the convergence of first order optimization algorithms to first and \textit{second} order stationary points under generalizations of smoothness. As a consequence, we establish the first convergence guarantees for first order methods to second order stationary points under generalizations of smoothness. We demonstrate that several canonical examples fall under our framework, and highlight practical implications. |
| title | Efficiently Escaping Saddle Points under Generalized Smoothness via Self-Bounding Regularity |
| topic | Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2503.04712 |