Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910973409886208 |
|---|---|
| author | Szatkowski, Filip Zheng, Yaoyue Yang, Fei Twardowski, Bartłomiej Trzciński, Tomasz van de Weijer, Joost |
| author_facet | Szatkowski, Filip Zheng, Yaoyue Yang, Fei Twardowski, Bartłomiej Trzciński, Tomasz van de Weijer, Joost |
| contents | Continual learning is crucial for applying machine learning in challenging, dynamic, and often resource-constrained environments. However, catastrophic forgetting - overwriting previously learned knowledge when new information is acquired - remains a major challenge. In this work, we examine the intermediate representations in neural network layers during continual learning and find that such representations are less prone to forgetting, highlighting their potential to accelerate computation. Motivated by these findings, we propose to use auxiliary classifiers(ACs) to enhance performance and demonstrate that integrating ACs into various continual learning methods consistently improves accuracy across diverse evaluation settings, yielding an average 10% relative gain. We also leverage the ACs to reduce the average cost of the inference by 10-60% without compromising accuracy, enabling the model to return the predictions before computing all the layers. Our approach provides a scalable and efficient solution for continual learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07404 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers Szatkowski, Filip Zheng, Yaoyue Yang, Fei Twardowski, Bartłomiej Trzciński, Tomasz van de Weijer, Joost Machine Learning Artificial Intelligence Continual learning is crucial for applying machine learning in challenging, dynamic, and often resource-constrained environments. However, catastrophic forgetting - overwriting previously learned knowledge when new information is acquired - remains a major challenge. In this work, we examine the intermediate representations in neural network layers during continual learning and find that such representations are less prone to forgetting, highlighting their potential to accelerate computation. Motivated by these findings, we propose to use auxiliary classifiers(ACs) to enhance performance and demonstrate that integrating ACs into various continual learning methods consistently improves accuracy across diverse evaluation settings, yielding an average 10% relative gain. We also leverage the ACs to reduce the average cost of the inference by 10-60% without compromising accuracy, enabling the model to return the predictions before computing all the layers. Our approach provides a scalable and efficient solution for continual learning. |
| title | Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.07404 |