Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2304.01239 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916617944825856 |
|---|---|
| author | Houyon, Joachim Cioppa, Anthony Ghunaim, Yasir Alfarra, Motasem Halin, Anaïs Henry, Maxim Ghanem, Bernard Van Droogenbroeck, Marc |
| author_facet | Houyon, Joachim Cioppa, Anthony Ghunaim, Yasir Alfarra, Motasem Halin, Anaïs Henry, Maxim Ghanem, Bernard Van Droogenbroeck, Marc |
| contents | In recent years, online distillation has emerged as a powerful technique for adapting real-time deep neural networks on the fly using a slow, but accurate teacher model. However, a major challenge in online distillation is catastrophic forgetting when the domain shifts, which occurs when the student model is updated with data from the new domain and forgets previously learned knowledge. In this paper, we propose a solution to this issue by leveraging the power of continual learning methods to reduce the impact of domain shifts. Specifically, we integrate several state-of-the-art continual learning methods in the context of online distillation and demonstrate their effectiveness in reducing catastrophic forgetting. Furthermore, we provide a detailed analysis of our proposed solution in the case of cyclic domain shifts. Our experimental results demonstrate the efficacy of our approach in improving the robustness and accuracy of online distillation, with potential applications in domains such as video surveillance or autonomous driving. Overall, our work represents an important step forward in the field of online distillation and continual learning, with the potential to significantly impact real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_01239 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Online Distillation with Continual Learning for Cyclic Domain Shifts Houyon, Joachim Cioppa, Anthony Ghunaim, Yasir Alfarra, Motasem Halin, Anaïs Henry, Maxim Ghanem, Bernard Van Droogenbroeck, Marc Computer Vision and Pattern Recognition Machine Learning In recent years, online distillation has emerged as a powerful technique for adapting real-time deep neural networks on the fly using a slow, but accurate teacher model. However, a major challenge in online distillation is catastrophic forgetting when the domain shifts, which occurs when the student model is updated with data from the new domain and forgets previously learned knowledge. In this paper, we propose a solution to this issue by leveraging the power of continual learning methods to reduce the impact of domain shifts. Specifically, we integrate several state-of-the-art continual learning methods in the context of online distillation and demonstrate their effectiveness in reducing catastrophic forgetting. Furthermore, we provide a detailed analysis of our proposed solution in the case of cyclic domain shifts. Our experimental results demonstrate the efficacy of our approach in improving the robustness and accuracy of online distillation, with potential applications in domains such as video surveillance or autonomous driving. Overall, our work represents an important step forward in the field of online distillation and continual learning, with the potential to significantly impact real-world applications. |
| title | Online Distillation with Continual Learning for Cyclic Domain Shifts |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2304.01239 |