InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917635361341440 |
|---|---|
| author | Dong, Xiaoyi Zhang, Pan Zang, Yuhang Cao, Yuhang Wang, Bin Ouyang, Linke Zhang, Songyang Duan, Haodong Zhang, Wenwei Li, Yining Yan, Hang Gao, Yang Chen, Zhe Zhang, Xinyue Li, Wei Li, Jingwen Wang, Wenhai Chen, Kai He, Conghui Zhang, Xingcheng Dai, Jifeng Qiao, Yu Lin, Dahua Wang, Jiaqi |
| author_facet | Dong, Xiaoyi Zhang, Pan Zang, Yuhang Cao, Yuhang Wang, Bin Ouyang, Linke Zhang, Songyang Duan, Haodong Zhang, Wenwei Li, Yining Yan, Hang Gao, Yang Chen, Zhe Zhang, Xinyue Li, Wei Li, Jingwen Wang, Wenhai Chen, Kai He, Conghui Zhang, Xingcheng Dai, Jifeng Qiao, Yu Lin, Dahua Wang, Jiaqi |
| contents | The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 x 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 x 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The InternLM-XComposer2-4KHD model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06512 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD Dong, Xiaoyi Zhang, Pan Zang, Yuhang Cao, Yuhang Wang, Bin Ouyang, Linke Zhang, Songyang Duan, Haodong Zhang, Wenwei Li, Yining Yan, Hang Gao, Yang Chen, Zhe Zhang, Xinyue Li, Wei Li, Jingwen Wang, Wenhai Chen, Kai He, Conghui Zhang, Xingcheng Dai, Jifeng Qiao, Yu Lin, Dahua Wang, Jiaqi Computer Vision and Pattern Recognition Computation and Language The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 x 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 x 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The InternLM-XComposer2-4KHD model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer. |
| title | InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2404.06512 |