MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912052267712512 |
|---|---|
| author | Zhang, Haotian Gao, Mingfei Gan, Zhe Dufter, Philipp Wenzel, Nina Huang, Forrest Shah, Dhruti Du, Xianzhi Zhang, Bowen Li, Yanghao Dodge, Sam You, Keen Yang, Zhen Timofeev, Aleksei Xu, Mingze Chen, Hong-You Fauconnier, Jean-Philippe Lai, Zhengfeng You, Haoxuan Wang, Zirui Dehghan, Afshin Grasch, Peter Yang, Yinfei |
| author_facet | Zhang, Haotian Gao, Mingfei Gan, Zhe Dufter, Philipp Wenzel, Nina Huang, Forrest Shah, Dhruti Du, Xianzhi Zhang, Bowen Li, Yanghao Dodge, Sam You, Keen Yang, Zhen Timofeev, Aleksei Xu, Mingze Chen, Hong-You Fauconnier, Jean-Philippe Lai, Zhengfeng You, Haoxuan Wang, Zirui Dehghan, Afshin Grasch, Peter Yang, Yinfei |
| contents | We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_20566 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning Zhang, Haotian Gao, Mingfei Gan, Zhe Dufter, Philipp Wenzel, Nina Huang, Forrest Shah, Dhruti Du, Xianzhi Zhang, Bowen Li, Yanghao Dodge, Sam You, Keen Yang, Zhen Timofeev, Aleksei Xu, Mingze Chen, Hong-You Fauconnier, Jean-Philippe Lai, Zhengfeng You, Haoxuan Wang, Zirui Dehghan, Afshin Grasch, Peter Yang, Yinfei Computer Vision and Pattern Recognition Computation and Language Machine Learning We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development. |
| title | MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2409.20566 |