When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914167288496128 |
|---|---|
| author | Tuchinda, Pume Pengpun, Parinthapat Chumpu, Romrawin Nutanong, Sarana Limkonchotiwat, Peerat |
| author_facet | Tuchinda, Pume Pengpun, Parinthapat Chumpu, Romrawin Nutanong, Sarana Limkonchotiwat, Peerat |
| contents | Vision-language models (VLMs) have achieved remarkable success across multimodal tasks, yet their substantial computational demands hinder efficient deployment. Knowledge distillation (KD) has emerged as a powerful approach for building lightweight but competitive models, with strong evidence from both language and vision domains. However, its application to VLMs, particularly CLIP-style models, remains limited, often constrained to small-scale teachers and narrow evaluation tasks such as classification or retrieval. In this work, we present the first systematic study of distillation across a range of CLIP-style teacher models, ranging from standard baselines to large-scale state-of-the-art models. Contrary to trends observed in NLP and vision, we find that stronger teachers do not consistently yield better students; in fact, existing distillation frameworks often fail to scale, leading to degraded performance in downstream multimodal tasks such as visual question answering. Our findings challenge prevailing assumptions in KD and point toward new directions for designing parameter-efficient multimodal models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_17886 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA Tuchinda, Pume Pengpun, Parinthapat Chumpu, Romrawin Nutanong, Sarana Limkonchotiwat, Peerat Computer Vision and Pattern Recognition Computation and Language Vision-language models (VLMs) have achieved remarkable success across multimodal tasks, yet their substantial computational demands hinder efficient deployment. Knowledge distillation (KD) has emerged as a powerful approach for building lightweight but competitive models, with strong evidence from both language and vision domains. However, its application to VLMs, particularly CLIP-style models, remains limited, often constrained to small-scale teachers and narrow evaluation tasks such as classification or retrieval. In this work, we present the first systematic study of distillation across a range of CLIP-style teacher models, ranging from standard baselines to large-scale state-of-the-art models. Contrary to trends observed in NLP and vision, we find that stronger teachers do not consistently yield better students; in fact, existing distillation frameworks often fail to scale, leading to degraded performance in downstream multimodal tasks such as visual question answering. Our findings challenge prevailing assumptions in KD and point toward new directions for designing parameter-efficient multimodal models. |
| title | When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2511.17886 |