Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.18558 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929659208269824 |
|---|---|
| author | Gu, Shuhao Zhang, Jialing Zhou, Siyuan Yu, Kevin Xing, Zhaohu Wang, Liangdong Cao, Zhou Jia, Jintao Zhang, Zhuoyi Wang, Yixuan Hu, Zhenchong Zhang, Bo-Wen Li, Jijie Liang, Dong Zhao, Yingli Wang, Songjing Ao, Yulong Ju, Yiming Ma, Huanhuan Li, Xiaotong Diao, Haiwen Cui, Yufeng Wang, Xinlong Liu, Yaoqi Feng, Fangxiang Liu, Guang |
| author_facet | Gu, Shuhao Zhang, Jialing Zhou, Siyuan Yu, Kevin Xing, Zhaohu Wang, Liangdong Cao, Zhou Jia, Jintao Zhang, Zhuoyi Wang, Yixuan Hu, Zhenchong Zhang, Bo-Wen Li, Jijie Liang, Dong Zhao, Yingli Wang, Songjing Ao, Yulong Ju, Yiming Ma, Huanhuan Li, Xiaotong Diao, Haiwen Cui, Yufeng Wang, Xinlong Liu, Yaoqi Feng, Fangxiang Liu, Guang |
| contents | Recently, Vision-Language Models (VLMs) have achieved remarkable progress in multimodal tasks, and multimodal instruction data serves as the foundation for enhancing VLM capabilities. Despite the availability of several open-source multimodal datasets, limitations in the scale and quality of open-source instruction data hinder the performance of VLMs trained on these datasets, leading to a significant gap compared to models trained on closed-source data. To address this challenge, we introduce Infinity-MM, a large-scale multimodal instruction dataset. We collected the available multimodal instruction datasets and performed unified preprocessing, resulting in a dataset with over 40 million samples that ensures diversity and accuracy. Furthermore, to enable large-scale expansion of instruction data and support the continuous acquisition of high-quality data, we propose a synthetic instruction generation method based on a tagging system and open-source VLMs. By establishing correspondences between different types of images and associated instruction types, this method can provide essential guidance during data synthesis. Leveraging this high-quality data, we have trained a 2-billion-parameter Vision-Language Model, Aquila-VL-2B, which achieves state-of-the-art (SOTA) performance among models of similar scale. The data is available at: https://huggingface.co/datasets/BAAI/Infinity-MM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18558 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data Gu, Shuhao Zhang, Jialing Zhou, Siyuan Yu, Kevin Xing, Zhaohu Wang, Liangdong Cao, Zhou Jia, Jintao Zhang, Zhuoyi Wang, Yixuan Hu, Zhenchong Zhang, Bo-Wen Li, Jijie Liang, Dong Zhao, Yingli Wang, Songjing Ao, Yulong Ju, Yiming Ma, Huanhuan Li, Xiaotong Diao, Haiwen Cui, Yufeng Wang, Xinlong Liu, Yaoqi Feng, Fangxiang Liu, Guang Computation and Language Recently, Vision-Language Models (VLMs) have achieved remarkable progress in multimodal tasks, and multimodal instruction data serves as the foundation for enhancing VLM capabilities. Despite the availability of several open-source multimodal datasets, limitations in the scale and quality of open-source instruction data hinder the performance of VLMs trained on these datasets, leading to a significant gap compared to models trained on closed-source data. To address this challenge, we introduce Infinity-MM, a large-scale multimodal instruction dataset. We collected the available multimodal instruction datasets and performed unified preprocessing, resulting in a dataset with over 40 million samples that ensures diversity and accuracy. Furthermore, to enable large-scale expansion of instruction data and support the continuous acquisition of high-quality data, we propose a synthetic instruction generation method based on a tagging system and open-source VLMs. By establishing correspondences between different types of images and associated instruction types, this method can provide essential guidance during data synthesis. Leveraging this high-quality data, we have trained a 2-billion-parameter Vision-Language Model, Aquila-VL-2B, which achieves state-of-the-art (SOTA) performance among models of similar scale. The data is available at: https://huggingface.co/datasets/BAAI/Infinity-MM. |
| title | Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.18558 |