Dispelling the Curse of Singularities in Neural Network Optimizations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910020826824704 |
|---|---|
| author | Cao, Hengjie Chen, Mengyi Yang, Yifeng Dong, Fang Huang, Ruijun Chen, Anrui Zhou, Jixian Dong, Mingzhi Wang, Yujiang Li, Dongsheng Fang, Wenyi Lin, Yuanyi Wu, Fan Shang, Li |
| author_facet | Cao, Hengjie Chen, Mengyi Yang, Yifeng Dong, Fang Huang, Ruijun Chen, Anrui Zhou, Jixian Dong, Mingzhi Wang, Yujiang Li, Dongsheng Fang, Wenyi Lin, Yuanyi Wu, Fan Shang, Li |
| contents | This work investigates the optimization instability of deep neural networks from a less-explored yet insightful perspective: the emergence and amplification of singularities in the parametric space. Our analysis reveals that parametric singularities inevitably grow with gradient updates and further intensify alignment with representations, leading to increased singularities in the representation space. We show that the gradient Frobenius norms are bounded by the top singular values of the weight matrices, and as training progresses, the mutually reinforcing growth of weight and representation singularities, termed the curse of singularities, relaxes these bounds, escalating the risk of sharp loss explosions. To counter this, we propose Parametric Singularity Smoothing (PSS), a lightweight, flexible, and effective method for smoothing the singular spectra of weight matrices. Extensive experiments across diverse datasets, architectures, and optimizers demonstrate that PSS mitigates instability, restores trainability even after failure, and improves both training efficiency and generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01308 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Dispelling the Curse of Singularities in Neural Network Optimizations Cao, Hengjie Chen, Mengyi Yang, Yifeng Dong, Fang Huang, Ruijun Chen, Anrui Zhou, Jixian Dong, Mingzhi Wang, Yujiang Li, Dongsheng Fang, Wenyi Lin, Yuanyi Wu, Fan Shang, Li Machine Learning Artificial Intelligence This work investigates the optimization instability of deep neural networks from a less-explored yet insightful perspective: the emergence and amplification of singularities in the parametric space. Our analysis reveals that parametric singularities inevitably grow with gradient updates and further intensify alignment with representations, leading to increased singularities in the representation space. We show that the gradient Frobenius norms are bounded by the top singular values of the weight matrices, and as training progresses, the mutually reinforcing growth of weight and representation singularities, termed the curse of singularities, relaxes these bounds, escalating the risk of sharp loss explosions. To counter this, we propose Parametric Singularity Smoothing (PSS), a lightweight, flexible, and effective method for smoothing the singular spectra of weight matrices. Extensive experiments across diverse datasets, architectures, and optimizers demonstrate that PSS mitigates instability, restores trainability even after failure, and improves both training efficiency and generalization. |
| title | Dispelling the Curse of Singularities in Neural Network Optimizations |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.01308 |