Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917604562567168 |
|---|---|
| author | Wang, Weizhi Mrini, Khalil Yang, Linjie Kumar, Sateesh Tian, Yu Yan, Xifeng Wang, Heng |
| author_facet | Wang, Weizhi Mrini, Khalil Yang, Linjie Kumar, Sateesh Tian, Yu Yan, Xifeng Wang, Heng |
| contents | We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g., CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_02677 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters Wang, Weizhi Mrini, Khalil Yang, Linjie Kumar, Sateesh Tian, Yu Yan, Xifeng Wang, Heng Computer Vision and Pattern Recognition Computation and Language We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g., CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter. |
| title | Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2403.02677 |