Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913691816951808 |
|---|---|
| author | Rodriguez-Opazo, Cristian Abbasnejad, Ehsan Teney, Damien Damirchi, Hamed Marrese-Taylor, Edison Hengel, Anton van den |
| author_facet | Rodriguez-Opazo, Cristian Abbasnejad, Ehsan Teney, Damien Damirchi, Hamed Marrese-Taylor, Edison Hengel, Anton van den |
| contents | Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various architectures, from vision transformers (ViTs) to convolutional networks (ResNets) have been trained with CLIP to serve as general solutions to diverse vision tasks. This paper explores the differences across various CLIP-trained vision backbones. Despite using the same data and training objective, we find that these architectures have notably different representations, different classification performance across datasets, and different robustness properties to certain types of image perturbations. Our findings indicate a remarkable possible synergy across backbones by leveraging their respective strengths. In principle, classification accuracy could be improved by over 40 percentage with an informed selection of the optimal backbone per test example.Using this insight, we develop a straightforward yet powerful approach to adaptively ensemble multiple backbones. The approach uses as few as one labeled example per class to tune the adaptive combination of backbones. On a large collection of datasets, the method achieves a remarkable increase in accuracy of up to 39.1% over the best single backbone, well beyond traditional ensembles |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17139 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling Rodriguez-Opazo, Cristian Abbasnejad, Ehsan Teney, Damien Damirchi, Hamed Marrese-Taylor, Edison Hengel, Anton van den Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various architectures, from vision transformers (ViTs) to convolutional networks (ResNets) have been trained with CLIP to serve as general solutions to diverse vision tasks. This paper explores the differences across various CLIP-trained vision backbones. Despite using the same data and training objective, we find that these architectures have notably different representations, different classification performance across datasets, and different robustness properties to certain types of image perturbations. Our findings indicate a remarkable possible synergy across backbones by leveraging their respective strengths. In principle, classification accuracy could be improved by over 40 percentage with an informed selection of the optimal backbone per test example.Using this insight, we develop a straightforward yet powerful approach to adaptively ensemble multiple backbones. The approach uses as few as one labeled example per class to tune the adaptive combination of backbones. On a large collection of datasets, the method achieves a remarkable increase in accuracy of up to 39.1% over the best single backbone, well beyond traditional ensembles |
| title | Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2405.17139 |