Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929399899619328 |
|---|---|
| author | Zhao, Lirui Zhang, Yuxin Chao, Fei Ji, Rongrong |
| author_facet | Zhao, Lirui Zhang, Yuxin Chao, Fei Ji, Rongrong |
| contents | The poor cross-architecture generalization of dataset distillation greatly weakens its practical significance. This paper attempts to mitigate this issue through an empirical study, which suggests that the synthetic datasets undergo an inductive bias towards the distillation model. Therefore, the evaluation model is strictly confined to having similar architectures of the distillation model. We propose a novel method of EvaLuation with distillation Feature (ELF), which utilizes features from intermediate layers of the distillation model for the cross-architecture evaluation. In this manner, the evaluation model learns from bias-free knowledge therefore its architecture becomes unfettered while retaining performance. By performing extensive experiments, we successfully prove that ELF can well enhance the cross-architecture generalization of current DD methods. Code of this project is at \url{https://github.com/Lirui-Zhao/ELF}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_05598 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study Zhao, Lirui Zhang, Yuxin Chao, Fei Ji, Rongrong Machine Learning The poor cross-architecture generalization of dataset distillation greatly weakens its practical significance. This paper attempts to mitigate this issue through an empirical study, which suggests that the synthetic datasets undergo an inductive bias towards the distillation model. Therefore, the evaluation model is strictly confined to having similar architectures of the distillation model. We propose a novel method of EvaLuation with distillation Feature (ELF), which utilizes features from intermediate layers of the distillation model for the cross-architecture evaluation. In this manner, the evaluation model learns from bias-free knowledge therefore its architecture becomes unfettered while retaining performance. By performing extensive experiments, we successfully prove that ELF can well enhance the cross-architecture generalization of current DD methods. Code of this project is at \url{https://github.com/Lirui-Zhao/ELF}. |
| title | Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2312.05598 |