De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913288768454656 |
|---|---|
| author | Wang, Yuzheng Yang, Dingkang Chen, Zhaoyu Liu, Yang Liu, Siao Zhang, Wenqiang Zhang, Lihua Qi, Lizhe |
| author_facet | Wang, Yuzheng Yang, Dingkang Chen, Zhaoyu Liu, Yang Liu, Siao Zhang, Wenqiang Zhang, Lihua Qi, Lizhe |
| contents | Data-Free Knowledge Distillation (DFKD) is a promising task to train high-performance small models to enhance actual deployment without relying on the original training data. Existing methods commonly avoid relying on private data by utilizing synthetic or sampled data. However, a long-overlooked issue is that the severe distribution shifts between their substitution and original data, which manifests as huge differences in the quality of images and class proportions. The harmful shifts are essentially the confounder that significantly causes performance bottlenecks. To tackle the issue, this paper proposes a novel perspective with causal inference to disentangle the student models from the impact of such shifts. By designing a customized causal graph, we first reveal the causalities among the variables in the DFKD task. Subsequently, we propose a Knowledge Distillation Causal Intervention (KDCI) framework based on the backdoor adjustment to de-confound the confounder. KDCI can be flexibly combined with most existing state-of-the-art baselines. Experiments in combination with six representative DFKD methods demonstrate the effectiveness of our KDCI, which can obviously help existing methods under almost all settings, \textit{e.g.}, improving the baseline by up to 15.54\% accuracy on the CIFAR-100 dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_19539 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts Wang, Yuzheng Yang, Dingkang Chen, Zhaoyu Liu, Yang Liu, Siao Zhang, Wenqiang Zhang, Lihua Qi, Lizhe Computer Vision and Pattern Recognition Data-Free Knowledge Distillation (DFKD) is a promising task to train high-performance small models to enhance actual deployment without relying on the original training data. Existing methods commonly avoid relying on private data by utilizing synthetic or sampled data. However, a long-overlooked issue is that the severe distribution shifts between their substitution and original data, which manifests as huge differences in the quality of images and class proportions. The harmful shifts are essentially the confounder that significantly causes performance bottlenecks. To tackle the issue, this paper proposes a novel perspective with causal inference to disentangle the student models from the impact of such shifts. By designing a customized causal graph, we first reveal the causalities among the variables in the DFKD task. Subsequently, we propose a Knowledge Distillation Causal Intervention (KDCI) framework based on the backdoor adjustment to de-confound the confounder. KDCI can be flexibly combined with most existing state-of-the-art baselines. Experiments in combination with six representative DFKD methods demonstrate the effectiveness of our KDCI, which can obviously help existing methods under almost all settings, \textit{e.g.}, improving the baseline by up to 15.54\% accuracy on the CIFAR-100 dataset. |
| title | De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.19539 |