On the Benefits of Weight Normalization for Overparameterized Matrix Sensing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912621188349952 |
|---|---|
| author | Wei, Yudong Zhang, Liang Li, Bingcong He, Niao |
| author_facet | Wei, Yudong Zhang, Liang Li, Bingcong He, Niao |
| contents | While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized) weight normalization (WN) applied to the overparameterized matrix sensing problem. We prove that WN with Riemannian optimization achieves linear convergence, yielding an exponential speedup over standard methods that do not use WN. Our analysis further demonstrates that both iteration and sample complexity improve polynomially as the level of overparameterization increases. To the best of our knowledge, this work provides the first characterization of how WN leverages overparameterization for faster convergence in matrix sensing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01175 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Benefits of Weight Normalization for Overparameterized Matrix Sensing Wei, Yudong Zhang, Liang Li, Bingcong He, Niao Machine Learning Signal Processing Optimization and Control While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized) weight normalization (WN) applied to the overparameterized matrix sensing problem. We prove that WN with Riemannian optimization achieves linear convergence, yielding an exponential speedup over standard methods that do not use WN. Our analysis further demonstrates that both iteration and sample complexity improve polynomially as the level of overparameterization increases. To the best of our knowledge, this work provides the first characterization of how WN leverages overparameterization for faster convergence in matrix sensing. |
| title | On the Benefits of Weight Normalization for Overparameterized Matrix Sensing |
| topic | Machine Learning Signal Processing Optimization and Control |
| url | https://arxiv.org/abs/2510.01175 |