Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916759808770048 |
|---|---|
| author | Wei, Wenbo Le, Nicholas Chong Jia Lai, Choy Heng Feng, Ling |
| author_facet | Wei, Wenbo Le, Nicholas Chong Jia Lai, Choy Heng Feng, Ling |
| contents | We observe a novel 'multiple-descent' phenomenon during the training process of LSTM, in which the test loss goes through long cycles of up and down trend multiple times after the model is overtrained. By carrying out asymptotic stability analysis of the models, we found that the cycles in test loss are closely associated with the phase transition process between order and chaos, and the local optimal epochs are consistently at the critical transition point between the two phases. More importantly, the global optimal epoch occurs at the first transition from order to chaos, where the 'width' of the 'edge of chaos' is the widest, allowing the best exploration of better weight configurations for learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20030 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions Wei, Wenbo Le, Nicholas Chong Jia Lai, Choy Heng Feng, Ling Machine Learning Artificial Intelligence Chaotic Dynamics Computational Physics We observe a novel 'multiple-descent' phenomenon during the training process of LSTM, in which the test loss goes through long cycles of up and down trend multiple times after the model is overtrained. By carrying out asymptotic stability analysis of the models, we found that the cycles in test loss are closely associated with the phase transition process between order and chaos, and the local optimal epochs are consistently at the critical transition point between the two phases. More importantly, the global optimal epoch occurs at the first transition from order to chaos, where the 'width' of the 'edge of chaos' is the widest, allowing the best exploration of better weight configurations for learning. |
| title | Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions |
| topic | Machine Learning Artificial Intelligence Chaotic Dynamics Computational Physics |
| url | https://arxiv.org/abs/2505.20030 |