Mitigating Forgetting in Continual Learning with Selective Gradient Projection
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911548656582656 |
|---|---|
| author | Singh, Anika Dhaulakhandi, Aayush Chopade, Varun Malipati, Likhith Martinez, David Zhu, Kevin |
| author_facet | Singh, Anika Dhaulakhandi, Aayush Chopade, Varun Malipati, Likhith Martinez, David Zhu, Kevin |
| contents | As neural networks are increasingly deployed in dynamic environments, they face the challenge of catastrophic forgetting, the tendency to overwrite previously learned knowledge when adapting to new tasks, resulting in severe performance degradation on earlier tasks. We propose Selective Forgetting-Aware Optimization (SFAO), a dynamic method that regulates gradient directions via cosine similarity and per-layer gating, enabling controlled forgetting while balancing plasticity and stability. SFAO selectively projects, accepts, or discards updates using a tunable mechanism with efficient Monte Carlo approximation. Experiments on standard continual learning benchmarks show that SFAO achieves competitive accuracy with markedly lower memory cost, a 90$\%$ reduction, and improved forgetting on MNIST datasets, making it suitable for resource-constrained scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_26671 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Mitigating Forgetting in Continual Learning with Selective Gradient Projection Singh, Anika Dhaulakhandi, Aayush Chopade, Varun Malipati, Likhith Martinez, David Zhu, Kevin Machine Learning Optimization and Control As neural networks are increasingly deployed in dynamic environments, they face the challenge of catastrophic forgetting, the tendency to overwrite previously learned knowledge when adapting to new tasks, resulting in severe performance degradation on earlier tasks. We propose Selective Forgetting-Aware Optimization (SFAO), a dynamic method that regulates gradient directions via cosine similarity and per-layer gating, enabling controlled forgetting while balancing plasticity and stability. SFAO selectively projects, accepts, or discards updates using a tunable mechanism with efficient Monte Carlo approximation. Experiments on standard continual learning benchmarks show that SFAO achieves competitive accuracy with markedly lower memory cost, a 90$\%$ reduction, and improved forgetting on MNIST datasets, making it suitable for resource-constrained scenarios. |
| title | Mitigating Forgetting in Continual Learning with Selective Gradient Projection |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2603.26671 |