Prioritize Alignment in Dataset Distillation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913543991853056 |
|---|---|
| author | Li, Zekai Guo, Ziyao Zhao, Wangbo Zhang, Tianle Cheng, Zhi-Qi Khaki, Samir Zhang, Kaipeng Sajedi, Ahmad Plataniotis, Konstantinos N Wang, Kai You, Yang |
| author_facet | Li, Zekai Guo, Ziyao Zhao, Wangbo Zhang, Tianle Cheng, Zhi-Qi Khaki, Samir Zhang, Kaipeng Sajedi, Ahmad Plataniotis, Konstantinos N Wang, Kai You, Yang |
| contents | Dataset Distillation aims to compress a large dataset into a significantly more compact, synthetic one without compromising the performance of the trained models. To achieve this, existing methods use the agent model to extract information from the target dataset and embed it into the distilled dataset. Consequently, the quality of extracted and embedded information determines the quality of the distilled dataset. In this work, we find that existing methods introduce misaligned information in both information extraction and embedding stages. To alleviate this, we propose Prioritize Alignment in Dataset Distillation (PAD), which aligns information from the following two perspectives. 1) We prune the target dataset according to the compressing ratio to filter the information that can be extracted by the agent model. 2) We use only deep layers of the agent model to perform the distillation to avoid excessively introducing low-level information. This simple strategy effectively filters out misaligned information and brings non-trivial improvement for mainstream matching-based distillation algorithms. Furthermore, built on trajectory matching, \textbf{PAD} achieves remarkable improvements on various benchmarks, achieving state-of-the-art performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03360 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Prioritize Alignment in Dataset Distillation Li, Zekai Guo, Ziyao Zhao, Wangbo Zhang, Tianle Cheng, Zhi-Qi Khaki, Samir Zhang, Kaipeng Sajedi, Ahmad Plataniotis, Konstantinos N Wang, Kai You, Yang Machine Learning Artificial Intelligence Dataset Distillation aims to compress a large dataset into a significantly more compact, synthetic one without compromising the performance of the trained models. To achieve this, existing methods use the agent model to extract information from the target dataset and embed it into the distilled dataset. Consequently, the quality of extracted and embedded information determines the quality of the distilled dataset. In this work, we find that existing methods introduce misaligned information in both information extraction and embedding stages. To alleviate this, we propose Prioritize Alignment in Dataset Distillation (PAD), which aligns information from the following two perspectives. 1) We prune the target dataset according to the compressing ratio to filter the information that can be extracted by the agent model. 2) We use only deep layers of the agent model to perform the distillation to avoid excessively introducing low-level information. This simple strategy effectively filters out misaligned information and brings non-trivial improvement for mainstream matching-based distillation algorithms. Furthermore, built on trajectory matching, \textbf{PAD} achieves remarkable improvements on various benchmarks, achieving state-of-the-art performance. |
| title | Prioritize Alignment in Dataset Distillation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2408.03360 |