TinyLLaVA: A Framework of Small-scale Large Multimodal Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929252497096704 |
|---|---|
| author | Zhou, Baichuan Hu, Ying Weng, Xi Jia, Junlong Luo, Jie Liu, Xien Wu, Ji Huang, Lei |
| author_facet | Zhou, Baichuan Hu, Ying Weng, Xi Jia, Junlong Luo, Jie Liu, Xien Wu, Ji Huang, Lei |
| contents | We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects of different vision encoders, connection modules, language models, training data and training recipes. Our extensive experiments showed that better quality of data combined with better training recipes, smaller LMMs can consistently achieve on-par performances compared to bigger LMMs. Under our framework, we train a family of small-scale LMMs. Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as baselines for future research in terms of data scaling, training setups and model selections. Our model weights and codes will be made public. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_14289 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TinyLLaVA: A Framework of Small-scale Large Multimodal Models Zhou, Baichuan Hu, Ying Weng, Xi Jia, Junlong Luo, Jie Liu, Xien Wu, Ji Huang, Lei Machine Learning Computation and Language We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects of different vision encoders, connection modules, language models, training data and training recipes. Our extensive experiments showed that better quality of data combined with better training recipes, smaller LMMs can consistently achieve on-par performances compared to bigger LMMs. Under our framework, we train a family of small-scale LMMs. Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as baselines for future research in terms of data scaling, training setups and model selections. Our model weights and codes will be made public. |
| title | TinyLLaVA: A Framework of Small-scale Large Multimodal Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2402.14289 |