Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.25525 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917530926317568 |
|---|---|
| author | Zhang, Mingxu Li, Yuhan Li, Lujundong Shen, Dazhong Xiong, Hui Sun, Ying |
| author_facet | Zhang, Mingxu Li, Yuhan Li, Lujundong Shen, Dazhong Xiong, Hui Sun, Ying |
| contents | Continual learning enables large language models to adapt to evolving tasks without retraining from scratch, yet catastrophic forgetting remains a central obstacle. Among continual learning methods, regularization-based approaches are widely used to constrain model updates and reduce forgetting, operating in weight space, gradient space, or output space. However, these dense representation spaces suffer from feature superposition, where multiple concepts are encoded in overlapping dimensions, making it difficult to selectively protect previously learned knowledge without impeding new-task learning. To address this issue, we propose \method (Sparse Autoencoder Feature Distillation), which anchors model representations in the sparse feature space of a pre-trained Sparse Autoencoder, where dense activations are decomposed into a sparse overcomplete basis that reduces representational entanglement, enabling more targeted regularization with less interference to new-task learning. Experiments on two continual learning benchmarks across three model architectures show that \method consistently outperforms existing regularization-based methods, achieving up to 52.70% average accuracy with only -0.46 backward transfer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25525 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Models Zhang, Mingxu Li, Yuhan Li, Lujundong Shen, Dazhong Xiong, Hui Sun, Ying Machine Learning Continual learning enables large language models to adapt to evolving tasks without retraining from scratch, yet catastrophic forgetting remains a central obstacle. Among continual learning methods, regularization-based approaches are widely used to constrain model updates and reduce forgetting, operating in weight space, gradient space, or output space. However, these dense representation spaces suffer from feature superposition, where multiple concepts are encoded in overlapping dimensions, making it difficult to selectively protect previously learned knowledge without impeding new-task learning. To address this issue, we propose \method (Sparse Autoencoder Feature Distillation), which anchors model representations in the sparse feature space of a pre-trained Sparse Autoencoder, where dense activations are decomposed into a sparse overcomplete basis that reduces representational entanglement, enabling more targeted regularization with less interference to new-task learning. Experiments on two continual learning benchmarks across three model architectures show that \method consistently outperforms existing regularization-based methods, achieving up to 52.70% average accuracy with only -0.46 backward transfer. |
| title | SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.25525 |