Latent Spectral Regularization for Continual Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917722441383936 |
|---|---|
| author | Frascaroli, Emanuele Benaglia, Riccardo Boschini, Matteo Moschella, Luca Fiorini, Cosimo Rodolà, Emanuele Calderara, Simone |
| author_facet | Frascaroli, Emanuele Benaglia, Riccardo Boschini, Matteo Moschella, Luca Fiorini, Cosimo Rodolà, Emanuele Calderara, Simone |
| contents | While biological intelligence grows organically as new knowledge is gathered throughout life, Artificial Neural Networks forget catastrophically whenever they face a changing training data distribution. Rehearsal-based Continual Learning (CL) approaches have been established as a versatile and reliable solution to overcome this limitation; however, sudden input disruptions and memory constraints are known to alter the consistency of their predictions. We study this phenomenon by investigating the geometric characteristics of the learner's latent space and find that replayed data points of different classes increasingly mix up, interfering with classification. Hence, we propose a geometric regularizer that enforces weak requirements on the Laplacian spectrum of the latent space, promoting a partitioning behavior. Our proposal, called Continual Spectral Regularizer for Incremental Learning (CaSpeR-IL), can be easily combined with any rehearsal-based CL approach and improves the performance of SOTA methods on standard benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2301_03345 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Latent Spectral Regularization for Continual Learning Frascaroli, Emanuele Benaglia, Riccardo Boschini, Matteo Moschella, Luca Fiorini, Cosimo Rodolà, Emanuele Calderara, Simone Machine Learning While biological intelligence grows organically as new knowledge is gathered throughout life, Artificial Neural Networks forget catastrophically whenever they face a changing training data distribution. Rehearsal-based Continual Learning (CL) approaches have been established as a versatile and reliable solution to overcome this limitation; however, sudden input disruptions and memory constraints are known to alter the consistency of their predictions. We study this phenomenon by investigating the geometric characteristics of the learner's latent space and find that replayed data points of different classes increasingly mix up, interfering with classification. Hence, we propose a geometric regularizer that enforces weak requirements on the Laplacian spectrum of the latent space, promoting a partitioning behavior. Our proposal, called Continual Spectral Regularizer for Incremental Learning (CaSpeR-IL), can be easily combined with any rehearsal-based CL approach and improves the performance of SOTA methods on standard benchmarks. |
| title | Latent Spectral Regularization for Continual Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2301.03345 |