Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2302.08484 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914706476761088 |
|---|---|
| author | Sivan, Hadar Gabel, Moshe Schuster, Assaf |
| author_facet | Sivan, Hadar Gabel, Moshe Schuster, Assaf |
| contents | Popular machine learning approaches forgo second-order information due to the difficulty of computing curvature in high dimensions. We present FOSI, a novel meta-algorithm that improves the performance of any base first-order optimizer by efficiently incorporating second-order information during the optimization process. In each iteration, FOSI implicitly splits the function into two quadratic functions defined on orthogonal subspaces, then uses a second-order method to minimize the first, and the base optimizer to minimize the other. We formally analyze FOSI's convergence and the conditions under which it improves a base optimizer. Our empirical evaluation demonstrates that FOSI improves the convergence rate and optimization time of first-order methods such as Heavy-Ball and Adam, and outperforms second-order methods (K-FAC and L-BFGS). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_08484 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | FOSI: Hybrid First and Second Order Optimization Sivan, Hadar Gabel, Moshe Schuster, Assaf Machine Learning Popular machine learning approaches forgo second-order information due to the difficulty of computing curvature in high dimensions. We present FOSI, a novel meta-algorithm that improves the performance of any base first-order optimizer by efficiently incorporating second-order information during the optimization process. In each iteration, FOSI implicitly splits the function into two quadratic functions defined on orthogonal subspaces, then uses a second-order method to minimize the first, and the base optimizer to minimize the other. We formally analyze FOSI's convergence and the conditions under which it improves a base optimizer. Our empirical evaluation demonstrates that FOSI improves the convergence rate and optimization time of first-order methods such as Heavy-Ball and Adam, and outperforms second-order methods (K-FAC and L-BFGS). |
| title | FOSI: Hybrid First and Second Order Optimization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2302.08484 |