Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917775312683008 |
|---|---|
| author | Beznosikov, Aleksandr Dobre, David Gidel, Gauthier |
| author_facet | Beznosikov, Aleksandr Dobre, David Gidel, Gauthier |
| contents | The Frank-Wolfe (FW) method is a popular approach for solving optimization problems with structured constraints that arise in machine learning applications. In recent years, stochastic versions of FW have gained popularity, motivated by large datasets for which the computation of the full gradient is prohibitively expensive. In this paper, we present two new variants of the FW algorithms for stochastic finite-sum minimization. Our algorithms have the best convergence guarantees of existing stochastic FW approaches for both convex and non-convex objective functions. Our methods do not have the issue of permanently collecting large batches, which is common to many stochastic projection-free approaches. Moreover, our second approach does not require either large batches or full deterministic gradients, which is a typical weakness of many techniques for finite-sum problems. The faster theoretical rates of our approaches are confirmed experimentally. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_11737 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features Beznosikov, Aleksandr Dobre, David Gidel, Gauthier Optimization and Control Machine Learning The Frank-Wolfe (FW) method is a popular approach for solving optimization problems with structured constraints that arise in machine learning applications. In recent years, stochastic versions of FW have gained popularity, motivated by large datasets for which the computation of the full gradient is prohibitively expensive. In this paper, we present two new variants of the FW algorithms for stochastic finite-sum minimization. Our algorithms have the best convergence guarantees of existing stochastic FW approaches for both convex and non-convex objective functions. Our methods do not have the issue of permanently collecting large batches, which is common to many stochastic projection-free approaches. Moreover, our second approach does not require either large batches or full deterministic gradients, which is a typical weakness of many techniques for finite-sum problems. The faster theoretical rates of our approaches are confirmed experimentally. |
| title | Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features |
| topic | Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2304.11737 |