VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912008480227328 |
|---|---|
| author | Zhang, Zaiwei Meyer, Gregory P. Lu, Zhichao Shrivastava, Ashish Ravichandran, Avinash Wolff, Eric M. |
| author_facet | Zhang, Zaiwei Meyer, Gregory P. Lu, Zhichao Shrivastava, Ashish Ravichandran, Avinash Wolff, Eric M. |
| contents | For visual recognition, knowledge distillation typically involves transferring knowledge from a large, well-trained teacher model to a smaller student model. In this paper, we introduce an effective method to distill knowledge from an off-the-shelf vision-language model (VLM), demonstrating that it provides novel supervision in addition to those from a conventional vision-only teacher model. Our key technical contribution is the development of a framework that generates novel text supervision and distills free-form text into a vision encoder. We showcase the effectiveness of our approach, termed VLM-KD, across various benchmark datasets, showing that it surpasses several state-of-the-art long-tail visual classifiers. To our knowledge, this work is the first to utilize knowledge distillation with text supervision generated by an off-the-shelf VLM and apply it to vanilla randomly initialized vision encoders. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16930 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition Zhang, Zaiwei Meyer, Gregory P. Lu, Zhichao Shrivastava, Ashish Ravichandran, Avinash Wolff, Eric M. Computer Vision and Pattern Recognition For visual recognition, knowledge distillation typically involves transferring knowledge from a large, well-trained teacher model to a smaller student model. In this paper, we introduce an effective method to distill knowledge from an off-the-shelf vision-language model (VLM), demonstrating that it provides novel supervision in addition to those from a conventional vision-only teacher model. Our key technical contribution is the development of a framework that generates novel text supervision and distills free-form text into a vision encoder. We showcase the effectiveness of our approach, termed VLM-KD, across various benchmark datasets, showing that it surpasses several state-of-the-art long-tail visual classifiers. To our knowledge, this work is the first to utilize knowledge distillation with text supervision generated by an off-the-shelf VLM and apply it to vanilla randomly initialized vision encoders. |
| title | VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.16930 |