Let ViT Speak: Generative Language-Image Pre-training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918477516767232 |
|---|---|
| author | Fang, Yan Lan, Mengcheng Huang, Zilong Lei, Weixian Zhao, Yunqing Zhong, Yujie Yu, Yingchen She, Qi Zhao, Yao Wei, Yunchao |
| author_facet | Fang, Yan Lan, Mengcheng Huang, Zilong Lei, Weixian Zhao, Yunqing Zhong, Yujie Yu, Yingchen She, Qi Zhao, Yao Wei, Yunchao |
| contents | In this paper, we present \textbf{Gen}erative \textbf{L}anguage-\textbf{I}mage \textbf{P}re-training (GenLIP), a minimalist generative pretraining framework for Vision Transformers (ViTs) designed for multimodal large language models (MLLMs). To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder. This design offers three key advantages: (1) \textbf{Simplicity}: a single transformer jointly models visual and textual tokens; (2) \textbf{Scalability}: it scales effectively with both data and model size; and (3) \textbf{Performance}: it achieves competitive or superior results across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data. After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_00809 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Let ViT Speak: Generative Language-Image Pre-training Fang, Yan Lan, Mengcheng Huang, Zilong Lei, Weixian Zhao, Yunqing Zhong, Yujie Yu, Yingchen She, Qi Zhao, Yao Wei, Yunchao Computer Vision and Pattern Recognition In this paper, we present \textbf{Gen}erative \textbf{L}anguage-\textbf{I}mage \textbf{P}re-training (GenLIP), a minimalist generative pretraining framework for Vision Transformers (ViTs) designed for multimodal large language models (MLLMs). To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder. This design offers three key advantages: (1) \textbf{Simplicity}: a single transformer jointly models visual and textual tokens; (2) \textbf{Scalability}: it scales effectively with both data and model size; and (3) \textbf{Performance}: it achieves competitive or superior results across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data. After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs. |
| title | Let ViT Speak: Generative Language-Image Pre-training |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.00809 |