Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.18965 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910743472898048 |
|---|---|
| author | Li, Bingcong Zhang, Liang Mokhtari, Aryan He, Niao |
| author_facet | Li, Bingcong Zhang, Liang Mokhtari, Aryan He, Niao |
| contents | This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth optimization. We introduce Nystrom initialization, which significantly improves the global convergence of Scaled Gradient Descent (ScaledGD) in both symmetric and asymmetric matrix factorization tasks. Specifically, we prove that ScaledGD with Nystrom initialization achieves quadratic convergence in cases where only linear rates were previously known. Furthermore, we extend this initialization to low-rank adapters (LoRA) commonly used for finetuning foundation models. Our approach, NoRA, i.e., LoRA with Nystrom initialization, demonstrates superior performance across various downstream tasks and model scales, from 1B to 7B parameters, in large language and diffusion models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18965 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Crucial Role of Initialization for Matrix Factorization Li, Bingcong Zhang, Liang Mokhtari, Aryan He, Niao Machine Learning Signal Processing Optimization and Control This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth optimization. We introduce Nystrom initialization, which significantly improves the global convergence of Scaled Gradient Descent (ScaledGD) in both symmetric and asymmetric matrix factorization tasks. Specifically, we prove that ScaledGD with Nystrom initialization achieves quadratic convergence in cases where only linear rates were previously known. Furthermore, we extend this initialization to low-rank adapters (LoRA) commonly used for finetuning foundation models. Our approach, NoRA, i.e., LoRA with Nystrom initialization, demonstrates superior performance across various downstream tasks and model scales, from 1B to 7B parameters, in large language and diffusion models. |
| title | On the Crucial Role of Initialization for Matrix Factorization |
| topic | Machine Learning Signal Processing Optimization and Control |
| url | https://arxiv.org/abs/2410.18965 |