iCD: A Implicit Clustering Distillation Mathod for Structural Information Mining
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908540892872704 |
|---|---|
| author | Xue, Xiang Ji, Yatu Ren, Qing-dao-er-ji Shi, Bao Lu, Min Wu, Nier Zhuang, Xufei Xu, Haiteng Cha, Gan-qi-qi-ge |
| author_facet | Xue, Xiang Ji, Yatu Ren, Qing-dao-er-ji Shi, Bao Lu, Min Wu, Nier Zhuang, Xufei Xu, Haiteng Cha, Gan-qi-qi-ge |
| contents | Logit Knowledge Distillation has gained substantial research interest in recent years due to its simplicity and lack of requirement for intermediate feature alignment; however, it suffers from limited interpretability in its decision-making process. To address this, we propose implicit Clustering Distillation (iCD): a simple and effective method that mines and transfers interpretable structural knowledge from logits, without requiring ground-truth labels or feature-space alignment. iCD leverages Gram matrices over decoupled local logit representations to enable student models to learn latent semantic structural patterns. Extensive experiments on benchmark datasets demonstrate the effectiveness of iCD across diverse teacher-student architectures, with particularly strong performance in fine-grained classification tasks -- achieving a peak improvement of +5.08% over the baseline. The code is available at: https://github.com/maomaochongaa/iCD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12553 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | iCD: A Implicit Clustering Distillation Mathod for Structural Information Mining Xue, Xiang Ji, Yatu Ren, Qing-dao-er-ji Shi, Bao Lu, Min Wu, Nier Zhuang, Xufei Xu, Haiteng Cha, Gan-qi-qi-ge Machine Learning Computer Vision and Pattern Recognition Logit Knowledge Distillation has gained substantial research interest in recent years due to its simplicity and lack of requirement for intermediate feature alignment; however, it suffers from limited interpretability in its decision-making process. To address this, we propose implicit Clustering Distillation (iCD): a simple and effective method that mines and transfers interpretable structural knowledge from logits, without requiring ground-truth labels or feature-space alignment. iCD leverages Gram matrices over decoupled local logit representations to enable student models to learn latent semantic structural patterns. Extensive experiments on benchmark datasets demonstrate the effectiveness of iCD across diverse teacher-student architectures, with particularly strong performance in fine-grained classification tasks -- achieving a peak improvement of +5.08% over the baseline. The code is available at: https://github.com/maomaochongaa/iCD. |
| title | iCD: A Implicit Clustering Distillation Mathod for Structural Information Mining |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.12553 |