Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909641907109888 |
|---|---|
| author | Yu, Lu Tao, Zhe Goswami, Dipam Yao, Hantao Twardowski, Bartłomiej Van de Weijer, Joost Xu, Changsheng |
| author_facet | Yu, Lu Tao, Zhe Goswami, Dipam Yao, Hantao Twardowski, Bartłomiej Van de Weijer, Joost Xu, Changsheng |
| contents | Deep neural networks (DNNs) excel on fixed datasets but struggle with incremental and shifting data in real-world scenarios. Continual learning addresses this challenge by allowing models to learn from new data while retaining previously learned knowledge. Existing methods mainly rely on visual features, often neglecting the rich semantic information encoded in text. The semantic knowledge available in the label information of the images, offers important semantic information that can be related with previously acquired knowledge of semantic classes. Consequently, effectively leveraging this information throughout continual learning is expected to be beneficial. To address this, we propose integrating semantic guidance within and across tasks by capturing semantic similarity using text embeddings. We start from a pre-trained CLIP model, employ the \emph{Semantically-guided Representation Learning (SG-RL)} module for a soft-assignment towards all current task classes, and use the Semantically-guided Knowledge Distillation (SG-KD) module for enhanced knowledge transfer. Experimental results demonstrate the superiority of our method on general and fine-grained datasets. Our code can be found in https://github.com/aprilsveryown/semantically-guided-continual-learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01076 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning Yu, Lu Tao, Zhe Goswami, Dipam Yao, Hantao Twardowski, Bartłomiej Van de Weijer, Joost Xu, Changsheng Computer Vision and Pattern Recognition Deep neural networks (DNNs) excel on fixed datasets but struggle with incremental and shifting data in real-world scenarios. Continual learning addresses this challenge by allowing models to learn from new data while retaining previously learned knowledge. Existing methods mainly rely on visual features, often neglecting the rich semantic information encoded in text. The semantic knowledge available in the label information of the images, offers important semantic information that can be related with previously acquired knowledge of semantic classes. Consequently, effectively leveraging this information throughout continual learning is expected to be beneficial. To address this, we propose integrating semantic guidance within and across tasks by capturing semantic similarity using text embeddings. We start from a pre-trained CLIP model, employ the \emph{Semantically-guided Representation Learning (SG-RL)} module for a soft-assignment towards all current task classes, and use the Semantically-guided Knowledge Distillation (SG-KD) module for enhanced knowledge transfer. Experimental results demonstrate the superiority of our method on general and fine-grained datasets. Our code can be found in https://github.com/aprilsveryown/semantically-guided-continual-learning. |
| title | Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.01076 |