AMD: Automatic Multi-step Distillation of Large-scale Vision Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929410411593728 |
|---|---|
| author | Han, Cheng Wang, Qifan Dianat, Sohail A. Rabbani, Majid Rao, Raghuveer M. Fang, Yi Guan, Qiang Huang, Lifu Liu, Dongfang |
| author_facet | Han, Cheng Wang, Qifan Dianat, Sohail A. Rabbani, Majid Rao, Raghuveer M. Fang, Yi Guan, Qiang Huang, Lifu Liu, Dongfang |
| contents | Transformer-based architectures have become the de-facto standard models for diverse vision tasks owing to their superior performance. As the size of the models continues to scale up, model distillation becomes extremely important in various real applications, particularly on devices limited by computational resources. However, prevailing knowledge distillation methods exhibit diminished efficacy when confronted with a large capacity gap between the teacher and the student, e.g, 10x compression rate. In this paper, we present a novel approach named Automatic Multi-step Distillation (AMD) for large-scale vision model compression. In particular, our distillation process unfolds across multiple steps. Initially, the teacher undergoes distillation to form an intermediate teacher-assistant model, which is subsequently distilled further to the student. An efficient and effective optimization framework is introduced to automatically identify the optimal teacher-assistant that leads to the maximal student performance. We conduct extensive experiments on multiple image classification datasets, including CIFAR-10, CIFAR-100, and ImageNet. The findings consistently reveal that our approach outperforms several established baselines, paving a path for future knowledge distillation methods on large-scale vision models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_04208 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AMD: Automatic Multi-step Distillation of Large-scale Vision Models Han, Cheng Wang, Qifan Dianat, Sohail A. Rabbani, Majid Rao, Raghuveer M. Fang, Yi Guan, Qiang Huang, Lifu Liu, Dongfang Computer Vision and Pattern Recognition Transformer-based architectures have become the de-facto standard models for diverse vision tasks owing to their superior performance. As the size of the models continues to scale up, model distillation becomes extremely important in various real applications, particularly on devices limited by computational resources. However, prevailing knowledge distillation methods exhibit diminished efficacy when confronted with a large capacity gap between the teacher and the student, e.g, 10x compression rate. In this paper, we present a novel approach named Automatic Multi-step Distillation (AMD) for large-scale vision model compression. In particular, our distillation process unfolds across multiple steps. Initially, the teacher undergoes distillation to form an intermediate teacher-assistant model, which is subsequently distilled further to the student. An efficient and effective optimization framework is introduced to automatically identify the optimal teacher-assistant that leads to the maximal student performance. We conduct extensive experiments on multiple image classification datasets, including CIFAR-10, CIFAR-100, and ImageNet. The findings consistently reveal that our approach outperforms several established baselines, paving a path for future knowledge distillation methods on large-scale vision models. |
| title | AMD: Automatic Multi-step Distillation of Large-scale Vision Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.04208 |