ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914836213923840 |
|---|---|
| author | Chen, Guiming Hardy Chen, Shunian Zhang, Ruifei Chen, Junying Wu, Xiangbo Zhang, Zhiyi Chen, Zhihong Li, Jianquan Wan, Xiang Wang, Benyou |
| author_facet | Chen, Guiming Hardy Chen, Shunian Zhang, Ruifei Chen, Junying Wu, Xiangbo Zhang, Zhiyi Chen, Zhihong Li, Jianquan Wan, Xiang Wang, Benyou |
| contents | Large vision-language models (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they require considerable computational resources for training and deployment. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To this end, we propose a comprehensive pipeline for generating a synthetic dataset. The key idea is to leverage strong proprietary models to generate (i) fine-grained image annotations for vision-language alignment and (ii) complex reasoning visual question-answering pairs for visual instruction fine-tuning, yielding 1.3M samples in total. We train a series of lite VLMs on the synthetic dataset and experimental results demonstrate the effectiveness of the proposed scheme, where they achieve competitive performance on 17 benchmarks among 4B LVLMs, and even perform on par with 7B/13B-scale models on various benchmarks. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. We name our dataset \textit{ALLaVA}, and open-source it to research community for developing better resource-efficient LVLMs for wider usage. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11684 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models Chen, Guiming Hardy Chen, Shunian Zhang, Ruifei Chen, Junying Wu, Xiangbo Zhang, Zhiyi Chen, Zhihong Li, Jianquan Wan, Xiang Wang, Benyou Computation and Language Artificial Intelligence Large vision-language models (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they require considerable computational resources for training and deployment. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To this end, we propose a comprehensive pipeline for generating a synthetic dataset. The key idea is to leverage strong proprietary models to generate (i) fine-grained image annotations for vision-language alignment and (ii) complex reasoning visual question-answering pairs for visual instruction fine-tuning, yielding 1.3M samples in total. We train a series of lite VLMs on the synthetic dataset and experimental results demonstrate the effectiveness of the proposed scheme, where they achieve competitive performance on 17 benchmarks among 4B LVLMs, and even perform on par with 7B/13B-scale models on various benchmarks. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. We name our dataset \textit{ALLaVA}, and open-source it to research community for developing better resource-efficient LVLMs for wider usage. |
| title | ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.11684 |