Active Data Curation Effectively Distills Large-Scale Multimodal Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915272116404224 |
|---|---|
| author | Udandarao, Vishaal Parthasarathy, Nikhil Naeem, Muhammad Ferjad Evans, Talfan Albanie, Samuel Tombari, Federico Xian, Yongqin Tonioni, Alessio Hénaff, Olivier J. |
| author_facet | Udandarao, Vishaal Parthasarathy, Nikhil Naeem, Muhammad Ferjad Evans, Talfan Albanie, Samuel Tombari, Federico Xian, Yongqin Tonioni, Alessio Hénaff, Olivier J. |
| contents | Knowledge distillation (KD) is the de facto standard for compressing large-scale models into smaller ones. Prior works have explored ever more complex KD strategies involving different objective functions, teacher-ensembles, and weight inheritance. In this work we explore an alternative, yet simple approach -- active data curation as effective distillation for contrastive multimodal pretraining. Our simple online batch selection method, ACID, outperforms strong KD baselines across various model-, data- and compute-configurations. Further, we find such an active data curation strategy to in fact be complementary to standard KD, and can be effectively combined to train highly performant inference-efficient models. Our simple and scalable pretraining framework, ACED, achieves state-of-the-art results across 27 zero-shot classification and retrieval tasks with upto 11% less inference FLOPs. We further demonstrate that our ACED models yield strong vision-encoders for training generative multimodal models in the LiT-Decoder setting, outperforming larger vision encoders for image-captioning and visual question-answering tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_18674 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Active Data Curation Effectively Distills Large-Scale Multimodal Models Udandarao, Vishaal Parthasarathy, Nikhil Naeem, Muhammad Ferjad Evans, Talfan Albanie, Samuel Tombari, Federico Xian, Yongqin Tonioni, Alessio Hénaff, Olivier J. Computer Vision and Pattern Recognition Machine Learning Knowledge distillation (KD) is the de facto standard for compressing large-scale models into smaller ones. Prior works have explored ever more complex KD strategies involving different objective functions, teacher-ensembles, and weight inheritance. In this work we explore an alternative, yet simple approach -- active data curation as effective distillation for contrastive multimodal pretraining. Our simple online batch selection method, ACID, outperforms strong KD baselines across various model-, data- and compute-configurations. Further, we find such an active data curation strategy to in fact be complementary to standard KD, and can be effectively combined to train highly performant inference-efficient models. Our simple and scalable pretraining framework, ACED, achieves state-of-the-art results across 27 zero-shot classification and retrieval tasks with upto 11% less inference FLOPs. We further demonstrate that our ACED models yield strong vision-encoders for training generative multimodal models in the LiT-Decoder setting, outperforming larger vision encoders for image-captioning and visual question-answering tasks. |
| title | Active Data Curation Effectively Distills Large-Scale Multimodal Models |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2411.18674 |