Distilling Invariant Representations with Dual Augmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908452164468736 |
|---|---|
| author | Giakoumoglou, Nikolaos Stathaki, Tania |
| author_facet | Giakoumoglou, Nikolaos Stathaki, Tania |
| contents | Knowledge distillation (KD) has been widely used to transfer knowledge from large, accurate models (teachers) to smaller, efficient ones (students). Recent methods have explored enforcing consistency by incorporating causal interpretations to distill invariant representations. In this work, we extend this line of research by introducing a dual augmentation strategy to promote invariant feature learning in both teacher and student models. Our approach leverages different augmentations applied to both models during distillation, pushing the student to capture robust, transferable features. This dual augmentation strategy complements invariant causal distillation by ensuring that the learned representations remain stable across a wider range of data variations and transformations. Extensive experiments on CIFAR-100 demonstrate the effectiveness of this approach, achieving competitive results in same-architecture KD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09474 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Distilling Invariant Representations with Dual Augmentation Giakoumoglou, Nikolaos Stathaki, Tania Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 68T07 I.4; I.2 Knowledge distillation (KD) has been widely used to transfer knowledge from large, accurate models (teachers) to smaller, efficient ones (students). Recent methods have explored enforcing consistency by incorporating causal interpretations to distill invariant representations. In this work, we extend this line of research by introducing a dual augmentation strategy to promote invariant feature learning in both teacher and student models. Our approach leverages different augmentations applied to both models during distillation, pushing the student to capture robust, transferable features. This dual augmentation strategy complements invariant causal distillation by ensuring that the learned representations remain stable across a wider range of data variations and transformations. Extensive experiments on CIFAR-100 demonstrate the effectiveness of this approach, achieving competitive results in same-architecture KD. |
| title | Distilling Invariant Representations with Dual Augmentation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 68T07 I.4; I.2 |
| url | https://arxiv.org/abs/2410.09474 |