POINTS: Improving Your Vision-language Model with Affordable Strategies
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910684773613568 |
|---|---|
| author | Liu, Yuan Zhao, Zhongyin Zhuang, Ziyuan Tian, Le Zhou, Xiao Zhou, Jie |
| author_facet | Liu, Yuan Zhao, Zhongyin Zhuang, Ziyuan Tian, Le Zhou, Xiao Zhou, Jie |
| contents | In recent years, vision-language models have made significant strides, excelling in tasks like optical character recognition and geometric problem-solving. However, several critical issues remain: 1) Proprietary models often lack transparency about their architectures, while open-source models need more detailed ablations of their training strategies. 2) Pre-training data in open-source works is under-explored, with datasets added empirically, making the process cumbersome. 3) Fine-tuning often focuses on adding datasets, leading to diminishing returns. To address these issues, we propose the following contributions: 1) We trained a robust baseline model using the latest advancements in vision-language models, introducing effective improvements and conducting comprehensive ablation and validation for each technique. 2) Inspired by recent work on large language models, we filtered pre-training data using perplexity, selecting the lowest perplexity data for training. This approach allowed us to train on a curated 1M dataset, achieving competitive performance. 3) During visual instruction tuning, we used model soup on different datasets when adding more datasets yielded marginal improvements. These innovations resulted in a 9B parameter model that performs competitively with state-of-the-art models. Our strategies are efficient and lightweight, making them easily adoptable by the community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_04828 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | POINTS: Improving Your Vision-language Model with Affordable Strategies Liu, Yuan Zhao, Zhongyin Zhuang, Ziyuan Tian, Le Zhou, Xiao Zhou, Jie Computer Vision and Pattern Recognition Artificial Intelligence Multimedia In recent years, vision-language models have made significant strides, excelling in tasks like optical character recognition and geometric problem-solving. However, several critical issues remain: 1) Proprietary models often lack transparency about their architectures, while open-source models need more detailed ablations of their training strategies. 2) Pre-training data in open-source works is under-explored, with datasets added empirically, making the process cumbersome. 3) Fine-tuning often focuses on adding datasets, leading to diminishing returns. To address these issues, we propose the following contributions: 1) We trained a robust baseline model using the latest advancements in vision-language models, introducing effective improvements and conducting comprehensive ablation and validation for each technique. 2) Inspired by recent work on large language models, we filtered pre-training data using perplexity, selecting the lowest perplexity data for training. This approach allowed us to train on a curated 1M dataset, achieving competitive performance. 3) During visual instruction tuning, we used model soup on different datasets when adding more datasets yielded marginal improvements. These innovations resulted in a 9B parameter model that performs competitively with state-of-the-art models. Our strategies are efficient and lightweight, making them easily adoptable by the community. |
| title | POINTS: Improving Your Vision-language Model with Affordable Strategies |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Multimedia |
| url | https://arxiv.org/abs/2409.04828 |