InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914657529233408 |
|---|---|
| author | Dong, Xiaoyi Zhang, Pan Zang, Yuhang Cao, Yuhang Wang, Bin Ouyang, Linke Wei, Xilin Zhang, Songyang Duan, Haodong Cao, Maosong Zhang, Wenwei Li, Yining Yan, Hang Gao, Yang Zhang, Xinyue Li, Wei Li, Jingwen Chen, Kai He, Conghui Zhang, Xingcheng Qiao, Yu Lin, Dahua Wang, Jiaqi |
| author_facet | Dong, Xiaoyi Zhang, Pan Zang, Yuhang Cao, Yuhang Wang, Bin Ouyang, Linke Wei, Xilin Zhang, Songyang Duan, Haodong Cao, Maosong Zhang, Wenwei Li, Yining Yan, Hang Gao, Yang Zhang, Xinyue Li, Wei Li, Jingwen Chen, Kai He, Conghui Zhang, Xingcheng Qiao, Yu Lin, Dahua Wang, Jiaqi |
| contents | We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension. This model goes beyond conventional vision-language understanding, adeptly crafting interleaved text-image content from diverse inputs like outlines, detailed textual specifications, and reference images, enabling highly customizable content creation. InternLM-XComposer2 proposes a Partial LoRA (PLoRA) approach that applies additional LoRA parameters exclusively to image tokens to preserve the integrity of pre-trained language knowledge, striking a balance between precise vision understanding and text composition with literary talent. Experimental results demonstrate the superiority of InternLM-XComposer2 based on InternLM2-7B in producing high-quality long-text multi-modal content and its exceptional vision-language understanding performance across various benchmarks, where it not only significantly outperforms existing multimodal models but also matches or even surpasses GPT-4V and Gemini Pro in certain assessments. This highlights its remarkable proficiency in the realm of multimodal understanding. The InternLM-XComposer2 model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_16420 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model Dong, Xiaoyi Zhang, Pan Zang, Yuhang Cao, Yuhang Wang, Bin Ouyang, Linke Wei, Xilin Zhang, Songyang Duan, Haodong Cao, Maosong Zhang, Wenwei Li, Yining Yan, Hang Gao, Yang Zhang, Xinyue Li, Wei Li, Jingwen Chen, Kai He, Conghui Zhang, Xingcheng Qiao, Yu Lin, Dahua Wang, Jiaqi Computer Vision and Pattern Recognition Computation and Language We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension. This model goes beyond conventional vision-language understanding, adeptly crafting interleaved text-image content from diverse inputs like outlines, detailed textual specifications, and reference images, enabling highly customizable content creation. InternLM-XComposer2 proposes a Partial LoRA (PLoRA) approach that applies additional LoRA parameters exclusively to image tokens to preserve the integrity of pre-trained language knowledge, striking a balance between precise vision understanding and text composition with literary talent. Experimental results demonstrate the superiority of InternLM-XComposer2 based on InternLM2-7B in producing high-quality long-text multi-modal content and its exceptional vision-language understanding performance across various benchmarks, where it not only significantly outperforms existing multimodal models but also matches or even surpasses GPT-4V and Gemini Pro in certain assessments. This highlights its remarkable proficiency in the realm of multimodal understanding. The InternLM-XComposer2 model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer. |
| title | InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2401.16420 |