MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916524362563584 |
|---|---|
| author | Liu, Yangzhou Cao, Yue Gao, Zhangwei Wang, Weiyun Chen, Zhe Wang, Wenhai Tian, Hao Lu, Lewei Zhu, Xizhou Lu, Tong Qiao, Yu Dai, Jifeng |
| author_facet | Liu, Yangzhou Cao, Yue Gao, Zhangwei Wang, Weiyun Chen, Zhe Wang, Wenhai Tian, Hao Lu, Lewei Zhu, Xizhou Lu, Tong Qiao, Yu Dai, Jifeng |
| contents | Despite the effectiveness of vision-language supervised fine-tuning in enhancing the performance of Vision Large Language Models (VLLMs). However, existing visual instruction tuning datasets include the following limitations: (1) Instruction annotation quality: despite existing VLLMs exhibiting strong performance, instructions generated by those advanced VLLMs may still suffer from inaccuracies, such as hallucinations. (2) Instructions and image diversity: the limited range of instruction types and the lack of diversity in image data may impact the model's ability to generate diversified and closer to real-world scenarios outputs. To address these challenges, we construct a high-quality, diverse visual instruction tuning dataset MMInstruct, which consists of 973K instructions from 24 domains. There are four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering and Short Visual Question Answering. To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3.5, and manual correction. Our instruction generation engine enables semi-automatic, low-cost, and multi-domain instruction generation at 1/6 the cost of manual construction. Through extensive experiment validation and ablation experiments, we demonstrate that MMInstruct could significantly improve the performance of VLLMs, e.g., the model fine-tuning on MMInstruct achieves new state-of-the-art performance on 10 out of 12 benchmarks. The code and data shall be available at https://github.com/yuecao0119/MMInstruct. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_15838 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity Liu, Yangzhou Cao, Yue Gao, Zhangwei Wang, Weiyun Chen, Zhe Wang, Wenhai Tian, Hao Lu, Lewei Zhu, Xizhou Lu, Tong Qiao, Yu Dai, Jifeng Computer Vision and Pattern Recognition Despite the effectiveness of vision-language supervised fine-tuning in enhancing the performance of Vision Large Language Models (VLLMs). However, existing visual instruction tuning datasets include the following limitations: (1) Instruction annotation quality: despite existing VLLMs exhibiting strong performance, instructions generated by those advanced VLLMs may still suffer from inaccuracies, such as hallucinations. (2) Instructions and image diversity: the limited range of instruction types and the lack of diversity in image data may impact the model's ability to generate diversified and closer to real-world scenarios outputs. To address these challenges, we construct a high-quality, diverse visual instruction tuning dataset MMInstruct, which consists of 973K instructions from 24 domains. There are four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering and Short Visual Question Answering. To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3.5, and manual correction. Our instruction generation engine enables semi-automatic, low-cost, and multi-domain instruction generation at 1/6 the cost of manual construction. Through extensive experiment validation and ablation experiments, we demonstrate that MMInstruct could significantly improve the performance of VLLMs, e.g., the model fine-tuning on MMInstruct achieves new state-of-the-art performance on 10 out of 12 benchmarks. The code and data shall be available at https://github.com/yuecao0119/MMInstruct. |
| title | MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.15838 |