Divide and not forget: Ensemble of selectively trained experts in Continual Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916165293441024 |
|---|---|
| author | Rypeść, Grzegorz Cygert, Sebastian Khan, Valeriya Trzciński, Tomasz Zieliński, Bartosz Twardowski, Bartłomiej |
| author_facet | Rypeść, Grzegorz Cygert, Sebastian Khan, Valeriya Trzciński, Tomasz Zieliński, Bartosz Twardowski, Bartłomiej |
| contents | Class-incremental learning is becoming more popular as it helps models widen their applicability while not forgetting what they already know. A trend in this area is to use a mixture-of-expert technique, where different models work together to solve the task. However, the experts are usually trained all at once using whole task data, which makes them all prone to forgetting and increasing computational burden. To address this limitation, we introduce a novel approach named SEED. SEED selects only one, the most optimal expert for a considered task, and uses data from this task to fine-tune only this expert. For this purpose, each expert represents each class with a Gaussian distribution, and the optimal expert is selected based on the similarity of those distributions. Consequently, SEED increases diversity and heterogeneity within the experts while maintaining the high stability of this ensemble method. The extensive experiments demonstrate that SEED achieves state-of-the-art performance in exemplar-free settings across various scenarios, showing the potential of expert diversification through data in continual learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_10191 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Divide and not forget: Ensemble of selectively trained experts in Continual Learning Rypeść, Grzegorz Cygert, Sebastian Khan, Valeriya Trzciński, Tomasz Zieliński, Bartosz Twardowski, Bartłomiej Machine Learning Computer Vision and Pattern Recognition Class-incremental learning is becoming more popular as it helps models widen their applicability while not forgetting what they already know. A trend in this area is to use a mixture-of-expert technique, where different models work together to solve the task. However, the experts are usually trained all at once using whole task data, which makes them all prone to forgetting and increasing computational burden. To address this limitation, we introduce a novel approach named SEED. SEED selects only one, the most optimal expert for a considered task, and uses data from this task to fine-tune only this expert. For this purpose, each expert represents each class with a Gaussian distribution, and the optimal expert is selected based on the similarity of those distributions. Consequently, SEED increases diversity and heterogeneity within the experts while maintaining the high stability of this ensemble method. The extensive experiments demonstrate that SEED achieves state-of-the-art performance in exemplar-free settings across various scenarios, showing the potential of expert diversification through data in continual learning. |
| title | Divide and not forget: Ensemble of selectively trained experts in Continual Learning |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.10191 |