Distillation-Guided Structural Transfer for Continual Learning Beyond Sparse Distributed Memory
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917151494897664 |
|---|---|
| author | Xue, Huiyan Ran, Xuming Li, Yaxin Xu, Qi Li, Enhui Xu, Yi Zhang, Qiang |
| author_facet | Xue, Huiyan Ran, Xuming Li, Yaxin Xu, Qi Li, Enhui Xu, Yi Zhang, Qiang |
| contents | Sparse neural systems are gaining traction for efficient continual learning due to their modularity and low interference. Architectures such as Sparse Distributed Memory Multi-Layer Perceptrons (SDMLP) construct task-specific subnetworks via Top-K activation and have shown resilience against catastrophic forgetting. However, their rigid modularity limits cross-task knowledge reuse and leads to performance degradation under high sparsity. We propose Selective Subnetwork Distillation (SSD), a structurally guided continual learning framework that treats distillation not as a regularizer but as a topology-aligned information conduit. SSD identifies neurons with high activation frequency and selectively distills knowledge within previous Top-K subnetworks and output logits, without requiring replay or task labels. This enables structural realignment while preserving sparse modularity. Experiments on Split CIFAR-10, CIFAR-100, and MNIST demonstrate that SSD improves accuracy, retention, and representation coverage, offering a structurally grounded solution for sparse continual learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_15267 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Distillation-Guided Structural Transfer for Continual Learning Beyond Sparse Distributed Memory Xue, Huiyan Ran, Xuming Li, Yaxin Xu, Qi Li, Enhui Xu, Yi Zhang, Qiang Machine Learning Sparse neural systems are gaining traction for efficient continual learning due to their modularity and low interference. Architectures such as Sparse Distributed Memory Multi-Layer Perceptrons (SDMLP) construct task-specific subnetworks via Top-K activation and have shown resilience against catastrophic forgetting. However, their rigid modularity limits cross-task knowledge reuse and leads to performance degradation under high sparsity. We propose Selective Subnetwork Distillation (SSD), a structurally guided continual learning framework that treats distillation not as a regularizer but as a topology-aligned information conduit. SSD identifies neurons with high activation frequency and selectively distills knowledge within previous Top-K subnetworks and output logits, without requiring replay or task labels. This enables structural realignment while preserving sparse modularity. Experiments on Split CIFAR-10, CIFAR-100, and MNIST demonstrate that SSD improves accuracy, retention, and representation coverage, offering a structurally grounded solution for sparse continual learning. |
| title | Distillation-Guided Structural Transfer for Continual Learning Beyond Sparse Distributed Memory |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.15267 |