Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.05391 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909032553381888 |
|---|---|
| author | Xia, Qianxin Du, Jiawei Zhang, Xin Zhang, Yuhan Wang, Jielei Lu, Guoming |
| author_facet | Xia, Qianxin Du, Jiawei Zhang, Xin Zhang, Yuhan Wang, Jielei Lu, Guoming |
| contents | Dataset distillation seeks to synthesize a highly compact dataset that achieves performance comparable to the original dataset on downstream tasks. For the classification task that use pre-trained self-supervised models as backbones, previous linear gradient matching optimizes synthetic images by encouraging them to mimic the gradient updates induced by real images on the linear classifier. However, this batch-level formulation requires loading thousands of real images and applying multiple rounds of differentiable augmentations to synthetic images at each distillation step, leading to substantial computational and memory overhead. In this paper, we introduce statistical flow matching , a stable and efficient supervised learning framework that optimizes synthetic images by aligning constant statistical flows from target class centers to non-target class centers in the original data. Our approach loads raw statistics only once and performs a single augmentation pass on the synthetic data, achieving performance comparable to or better than the state-of-the-art methods with 10x lower GPU memory usage and 4x shorter runtime. Furthermore, we propose a classifier inheritance strategy that reuses the classifier trained on the original dataset for inference, requiring only an extremely lightweight linear projector and marginal storage while achieving substantial performance gains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_05391 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching Xia, Qianxin Du, Jiawei Zhang, Xin Zhang, Yuhan Wang, Jielei Lu, Guoming Computer Vision and Pattern Recognition Dataset distillation seeks to synthesize a highly compact dataset that achieves performance comparable to the original dataset on downstream tasks. For the classification task that use pre-trained self-supervised models as backbones, previous linear gradient matching optimizes synthetic images by encouraging them to mimic the gradient updates induced by real images on the linear classifier. However, this batch-level formulation requires loading thousands of real images and applying multiple rounds of differentiable augmentations to synthetic images at each distillation step, leading to substantial computational and memory overhead. In this paper, we introduce statistical flow matching , a stable and efficient supervised learning framework that optimizes synthetic images by aligning constant statistical flows from target class centers to non-target class centers in the original data. Our approach loads raw statistics only once and performs a single augmentation pass on the synthetic data, achieving performance comparable to or better than the state-of-the-art methods with 10x lower GPU memory usage and 4x shorter runtime. Furthermore, we propose a classifier inheritance strategy that reuses the classifier trained on the original dataset for inference, requiring only an extremely lightweight linear projector and marginal storage while achieving substantial performance gains. |
| title | Efficient Dataset Distillation for Pre-Trained Self-Supervised Models via Statistical Flow Matching |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.05391 |