Convergence Conditions for Stochastic Line Search Based Optimization of Over-parametrized Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909769419194368 |
|---|---|
| author | Lapucci, Matteo Pucci, Davide |
| author_facet | Lapucci, Matteo Pucci, Davide |
| contents | In this paper, we deal with algorithms to solve the finite-sum problems related to fitting over-parametrized models, that typically satisfy the interpolation condition. In particular, we focus on approaches based on stochastic line searches and employing general search directions. We define conditions on the sequence of search directions that guarantee finite termination and bounds for the backtracking procedure. Moreover, we shed light on the additional property of directions needed to prove fast (linear) convergence of the general class of algorithms when applied to PL functions in the interpolation regime. From the point of view of algorithms design, the proposed analysis identifies safeguarding conditions that could be employed in relevant algorithmic frameworks. In particular, it could be of interest to integrate stochastic line searches within momentum, conjugate gradient or adaptive preconditioning methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03199 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Convergence Conditions for Stochastic Line Search Based Optimization of Over-parametrized Models Lapucci, Matteo Pucci, Davide Optimization and Control Machine Learning 90C30, 90C26, 90C06, 65K05, 68T07 In this paper, we deal with algorithms to solve the finite-sum problems related to fitting over-parametrized models, that typically satisfy the interpolation condition. In particular, we focus on approaches based on stochastic line searches and employing general search directions. We define conditions on the sequence of search directions that guarantee finite termination and bounds for the backtracking procedure. Moreover, we shed light on the additional property of directions needed to prove fast (linear) convergence of the general class of algorithms when applied to PL functions in the interpolation regime. From the point of view of algorithms design, the proposed analysis identifies safeguarding conditions that could be employed in relevant algorithmic frameworks. In particular, it could be of interest to integrate stochastic line searches within momentum, conjugate gradient or adaptive preconditioning methods. |
| title | Convergence Conditions for Stochastic Line Search Based Optimization of Over-parametrized Models |
| topic | Optimization and Control Machine Learning 90C30, 90C26, 90C06, 65K05, 68T07 |
| url | https://arxiv.org/abs/2408.03199 |