Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911691272355840 |
|---|---|
| author | Joudaki, Amir Lanzillotta, Giulia Razlighi, Mohammad Samragh Mirzadeh, Iman Alizadeh, Keivan Hofmann, Thomas Farajtabar, Mehrdad Faghri, Fartash |
| author_facet | Joudaki, Amir Lanzillotta, Giulia Razlighi, Mohammad Samragh Mirzadeh, Iman Alizadeh, Keivan Hofmann, Thomas Farajtabar, Mehrdad Faghri, Fartash |
| contents | Deep learning models excel in stationary data but struggle in non-stationary environments due to a phenomenon known as loss of plasticity (LoP), the degradation of their ability to learn in the future. This work presents a first-principles investigation of LoP in gradient-based learning. Grounded in dynamical systems theory, we formally define LoP by identifying stable manifolds in the parameter space that trap gradient trajectories. Our analysis reveals two primary mechanisms that create these traps: frozen units from activation saturation and cloned-unit manifolds from representational redundancy. Our framework uncovers a fundamental tension: properties that promote generalization in static settings, such as low-rank representations and simplicity biases, directly contribute to LoP in continual learning scenarios. We validate our theoretical analysis with numerical simulations and explore architectural choices or targeted perturbations as potential mitigation strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00304 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity Joudaki, Amir Lanzillotta, Giulia Razlighi, Mohammad Samragh Mirzadeh, Iman Alizadeh, Keivan Hofmann, Thomas Farajtabar, Mehrdad Faghri, Fartash Machine Learning Artificial Intelligence Deep learning models excel in stationary data but struggle in non-stationary environments due to a phenomenon known as loss of plasticity (LoP), the degradation of their ability to learn in the future. This work presents a first-principles investigation of LoP in gradient-based learning. Grounded in dynamical systems theory, we formally define LoP by identifying stable manifolds in the parameter space that trap gradient trajectories. Our analysis reveals two primary mechanisms that create these traps: frozen units from activation saturation and cloned-unit manifolds from representational redundancy. Our framework uncovers a fundamental tension: properties that promote generalization in static settings, such as low-rank representations and simplicity biases, directly contribute to LoP in continual learning scenarios. We validate our theoretical analysis with numerical simulations and explore architectural choices or targeted perturbations as potential mitigation strategies. |
| title | Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.00304 |