Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.06199 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916174464286720 |
|---|---|
| author | Zhu, Minjie Zhu, Yichen Liu, Xin Liu, Ning Xu, Zhiyuan Shen, Chaomin Peng, Yaxin Ou, Zhicai Feng, Feifei Tang, Jian |
| author_facet | Zhu, Minjie Zhu, Yichen Liu, Xin Liu, Ning Xu, Zhiyuan Shen, Chaomin Peng, Yaxin Ou, Zhicai Feng, Feifei Tang, Jian |
| contents | Multimodal Large Language Models (MLLMs) have showcased impressive skills in tasks related to visual understanding and reasoning. Yet, their widespread application faces obstacles due to the high computational demands during both the training and inference phases, restricting their use to a limited audience within the research and user communities. In this paper, we investigate the design aspects of Multimodal Small Language Models (MSLMs) and propose an efficient multimodal assistant named Mipha, which is designed to create synergy among various aspects: visual representation, language models, and optimization strategies. We show that without increasing the volume of training data, our Mipha-3B outperforms the state-of-the-art large MLLMs, especially LLaVA-1.5-13B, on multiple benchmarks. Through detailed discussion, we provide insights and guidelines for developing strong MSLMs that rival the capabilities of MLLMs. Our code is available at https://github.com/zhuyiche/llava-phi. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_06199 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models Zhu, Minjie Zhu, Yichen Liu, Xin Liu, Ning Xu, Zhiyuan Shen, Chaomin Peng, Yaxin Ou, Zhicai Feng, Feifei Tang, Jian Computer Vision and Pattern Recognition Computation and Language Multimodal Large Language Models (MLLMs) have showcased impressive skills in tasks related to visual understanding and reasoning. Yet, their widespread application faces obstacles due to the high computational demands during both the training and inference phases, restricting their use to a limited audience within the research and user communities. In this paper, we investigate the design aspects of Multimodal Small Language Models (MSLMs) and propose an efficient multimodal assistant named Mipha, which is designed to create synergy among various aspects: visual representation, language models, and optimization strategies. We show that without increasing the volume of training data, our Mipha-3B outperforms the state-of-the-art large MLLMs, especially LLaVA-1.5-13B, on multiple benchmarks. Through detailed discussion, we provide insights and guidelines for developing strong MSLMs that rival the capabilities of MLLMs. Our code is available at https://github.com/zhuyiche/llava-phi. |
| title | Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2403.06199 |