Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2510.01175 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912621188349952 |
|---|---|
| author | Wei, Yudong Zhang, Liang Li, Bingcong He, Niao |
| author_facet | Wei, Yudong Zhang, Liang Li, Bingcong He, Niao |
| contents | While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized) weight normalization (WN) applied to the overparameterized matrix sensing problem. We prove that WN with Riemannian optimization achieves linear convergence, yielding an exponential speedup over standard methods that do not use WN. Our analysis further demonstrates that both iteration and sample complexity improve polynomially as the level of overparameterization increases. To the best of our knowledge, this work provides the first characterization of how WN leverages overparameterization for faster convergence in matrix sensing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01175 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Benefits of Weight Normalization for Overparameterized Matrix Sensing Wei, Yudong Zhang, Liang Li, Bingcong He, Niao Machine Learning Signal Processing Optimization and Control While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized) weight normalization (WN) applied to the overparameterized matrix sensing problem. We prove that WN with Riemannian optimization achieves linear convergence, yielding an exponential speedup over standard methods that do not use WN. Our analysis further demonstrates that both iteration and sample complexity improve polynomially as the level of overparameterization increases. To the best of our knowledge, this work provides the first characterization of how WN leverages overparameterization for faster convergence in matrix sensing. |
| title | On the Benefits of Weight Normalization for Overparameterized Matrix Sensing |
| topic | Machine Learning Signal Processing Optimization and Control |
| url | https://arxiv.org/abs/2510.01175 |