Should VLMs be Pre-trained with Image Data?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909532824797184 |
|---|---|
| author | Keh, Sedrick Mercat, Jean Gadre, Samir Yitzhak Arora, Kushal Vasiljevic, Igor Burchfiel, Benjamin Song, Shuran Tedrake, Russ Kollar, Thomas Schmidt, Ludwig Dave, Achal |
| author_facet | Keh, Sedrick Mercat, Jean Gadre, Samir Yitzhak Arora, Kushal Vasiljevic, Igor Burchfiel, Benjamin Song, Shuran Tedrake, Russ Kollar, Thomas Schmidt, Ludwig Dave, Achal |
| contents | Pre-trained LLMs that are further trained with image data perform well on vision-language tasks. While adding images during a second training phase effectively unlocks this capability, it is unclear how much of a gain or loss this two-step pipeline gives over VLMs which integrate images earlier into the training process. To investigate this, we train models spanning various datasets, scales, image-text ratios, and amount of pre-training done before introducing vision tokens. We then fine-tune these models and evaluate their downstream performance on a suite of vision-language and text-only tasks. We find that pre-training with a mixture of image and text data allows models to perform better on vision-language tasks while maintaining strong performance on text-only evaluations. On an average of 6 diverse tasks, we find that for a 1B model, introducing visual tokens 80% of the way through pre-training results in a 2% average improvement over introducing visual tokens to a fully pre-trained model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_07603 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Should VLMs be Pre-trained with Image Data? Keh, Sedrick Mercat, Jean Gadre, Samir Yitzhak Arora, Kushal Vasiljevic, Igor Burchfiel, Benjamin Song, Shuran Tedrake, Russ Kollar, Thomas Schmidt, Ludwig Dave, Achal Computer Vision and Pattern Recognition Pre-trained LLMs that are further trained with image data perform well on vision-language tasks. While adding images during a second training phase effectively unlocks this capability, it is unclear how much of a gain or loss this two-step pipeline gives over VLMs which integrate images earlier into the training process. To investigate this, we train models spanning various datasets, scales, image-text ratios, and amount of pre-training done before introducing vision tokens. We then fine-tune these models and evaluate their downstream performance on a suite of vision-language and text-only tasks. We find that pre-training with a mixture of image and text data allows models to perform better on vision-language tasks while maintaining strong performance on text-only evaluations. On an average of 6 diverse tasks, we find that for a 1B model, introducing visual tokens 80% of the way through pre-training results in a 2% average improvement over introducing visual tokens to a fully pre-trained model. |
| title | Should VLMs be Pre-trained with Image Data? |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.07603 |