Implicit Regularization Makes Overparameterized Asymmetric Matrix Sensing Robust to Perturbations
Fuente:
arXiv
Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912547548954624 |
|---|---|
| author | Wind, Johan S. |
| author_facet | Wind, Johan S. |
| contents | Several key questions remain unanswered regarding overparameterized learning models. It is unclear how (stochastic) gradient descent finds solutions that generalize well, and in particular the role of small random initializations. Matrix sensing, which is the problem of reconstructing a low-rank matrix from a few linear measurements, has become a standard prototypical setting to study these phenomena. Previous works have shown that matrix sensing can be solved by factorized gradient descent, provided the random initialization is extremely small.
In this paper, we find that factorized gradient descent is highly robust to certain perturbations. This lets us use a perturbation term to capture both the effects of imperfect measurements, discretization by gradient descent, and other noise, resulting in a general formulation which we call \textit{perturbed gradient flow}. We find that not only is this equivalent formulation easier to work with, but it leads to sharper sample and time complexities than previous work, handles moderately small initializations, and the results are naturally robust to perturbations such as noisy measurements or changing measurement matrices. Finally, we also analyze mini-batch stochastic gradient descent using the formulation, where we find improved sample complexity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_01796 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Implicit Regularization Makes Overparameterized Asymmetric Matrix Sensing Robust to Perturbations Wind, Johan S. Machine Learning Optimization and Control Several key questions remain unanswered regarding overparameterized learning models. It is unclear how (stochastic) gradient descent finds solutions that generalize well, and in particular the role of small random initializations. Matrix sensing, which is the problem of reconstructing a low-rank matrix from a few linear measurements, has become a standard prototypical setting to study these phenomena. Previous works have shown that matrix sensing can be solved by factorized gradient descent, provided the random initialization is extremely small. In this paper, we find that factorized gradient descent is highly robust to certain perturbations. This lets us use a perturbation term to capture both the effects of imperfect measurements, discretization by gradient descent, and other noise, resulting in a general formulation which we call \textit{perturbed gradient flow}. We find that not only is this equivalent formulation easier to work with, but it leads to sharper sample and time complexities than previous work, handles moderately small initializations, and the results are naturally robust to perturbations such as noisy measurements or changing measurement matrices. Finally, we also analyze mini-batch stochastic gradient descent using the formulation, where we find improved sample complexity. |
| title | Implicit Regularization Makes Overparameterized Asymmetric Matrix Sensing Robust to Perturbations |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2309.01796 |