Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.16821 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910428194406400 |
|---|---|
| author | Chen, Zhe Wang, Weiyun Tian, Hao Ye, Shenglong Gao, Zhangwei Cui, Erfei Tong, Wenwen Hu, Kongzhi Luo, Jiapeng Ma, Zheng Ma, Ji Wang, Jiaqi Dong, Xiaoyi Yan, Hang Guo, Hewei He, Conghui Shi, Botian Jin, Zhenjiang Xu, Chao Wang, Bin Wei, Xingjian Li, Wei Zhang, Wenjian Zhang, Bo Cai, Pinlong Wen, Licheng Yan, Xiangchao Dou, Min Lu, Lewei Zhu, Xizhou Lu, Tong Lin, Dahua Qiao, Yu Dai, Jifeng Wang, Wenhai |
| author_facet | Chen, Zhe Wang, Weiyun Tian, Hao Ye, Shenglong Gao, Zhangwei Cui, Erfei Tong, Wenwen Hu, Kongzhi Luo, Jiapeng Ma, Zheng Ma, Ji Wang, Jiaqi Dong, Xiaoyi Yan, Hang Guo, Hewei He, Conghui Shi, Botian Jin, Zhenjiang Xu, Chao Wang, Bin Wei, Xingjian Li, Wei Zhang, Wenjian Zhang, Bo Cai, Pinlong Wen, Licheng Yan, Xiangchao Dou, Min Lu, Lewei Zhu, Xizhou Lu, Tong Lin, Dahua Qiao, Yu Dai, Jifeng Wang, Wenhai |
| contents | In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements: (1) Strong Vision Encoder: we explored a continuous learning strategy for the large-scale vision foundation model -- InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic High-Resolution: we divide images into tiles ranging from 1 to 40 of 448$\times$448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) High-Quality Bilingual Dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in OCR- and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks. Code has been released at https://github.com/OpenGVLab/InternVL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_16821 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites Chen, Zhe Wang, Weiyun Tian, Hao Ye, Shenglong Gao, Zhangwei Cui, Erfei Tong, Wenwen Hu, Kongzhi Luo, Jiapeng Ma, Zheng Ma, Ji Wang, Jiaqi Dong, Xiaoyi Yan, Hang Guo, Hewei He, Conghui Shi, Botian Jin, Zhenjiang Xu, Chao Wang, Bin Wei, Xingjian Li, Wei Zhang, Wenjian Zhang, Bo Cai, Pinlong Wen, Licheng Yan, Xiangchao Dou, Min Lu, Lewei Zhu, Xizhou Lu, Tong Lin, Dahua Qiao, Yu Dai, Jifeng Wang, Wenhai Computer Vision and Pattern Recognition In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements: (1) Strong Vision Encoder: we explored a continuous learning strategy for the large-scale vision foundation model -- InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic High-Resolution: we divide images into tiles ranging from 1 to 40 of 448$\times$448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) High-Quality Bilingual Dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in OCR- and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks. Code has been released at https://github.com/OpenGVLab/InternVL. |
| title | How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.16821 |