Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916922577125376 |
|---|---|
| author | Ersoy, Ibrahim Talha Wiesner, Karoline |
| author_facet | Ersoy, Ibrahim Talha Wiesner, Karoline |
| contents | Increasing the L2 regularization of Deep Neural Networks (DNNs) causes a first-order phase transition into the under-parametrized phase -- the so-called onset-of learning. We explain this transition via the scalar (Ricci) curvature of the error landscape. We predict new transition points as the data complexity is increased and, in accordance with the theory of phase transitions, the existence of hysteresis effects. We confirm both predictions numerically. Our results provide a natural explanation of the recently discovered phenomenon of '\emph{grokking}' as DNN models getting stuck in a local minimum of the error surface, corresponding to a lower accuracy phase. Our work paves the way for new probing methods of the intrinsic structure of DNNs in and beyond the L2 context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_06597 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks Ersoy, Ibrahim Talha Wiesner, Karoline Machine Learning Disordered Systems and Neural Networks Statistical Mechanics Data Analysis, Statistics and Probability Increasing the L2 regularization of Deep Neural Networks (DNNs) causes a first-order phase transition into the under-parametrized phase -- the so-called onset-of learning. We explain this transition via the scalar (Ricci) curvature of the error landscape. We predict new transition points as the data complexity is increased and, in accordance with the theory of phase transitions, the existence of hysteresis effects. We confirm both predictions numerically. Our results provide a natural explanation of the recently discovered phenomenon of '\emph{grokking}' as DNN models getting stuck in a local minimum of the error surface, corresponding to a lower accuracy phase. Our work paves the way for new probing methods of the intrinsic structure of DNNs in and beyond the L2 context. |
| title | Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks |
| topic | Machine Learning Disordered Systems and Neural Networks Statistical Mechanics Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2505.06597 |