Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916422331924480 |
|---|---|
| author | Lai, Zhengfeng Saveris, Vasileios Chen, Chen Chen, Hong-You Zhang, Haotian Zhang, Bowen Tebar, Juan Lao Hu, Wenze Gan, Zhe Grasch, Peter Cao, Meng Yang, Yinfei |
| author_facet | Lai, Zhengfeng Saveris, Vasileios Chen, Chen Chen, Hong-You Zhang, Haotian Zhang, Bowen Tebar, Juan Lao Hu, Wenze Gan, Zhe Grasch, Peter Cao, Meng Yang, Yinfei |
| contents | Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they can fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats. This comprehensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02740 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models Lai, Zhengfeng Saveris, Vasileios Chen, Chen Chen, Hong-You Zhang, Haotian Zhang, Bowen Tebar, Juan Lao Hu, Wenze Gan, Zhe Grasch, Peter Cao, Meng Yang, Yinfei Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they can fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats. This comprehensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models. |
| title | Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2410.02740 |