BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916478162305024 |
|---|---|
| author | Awadalla, Anas Xue, Le Shu, Manli Yan, An Wang, Jun Purushwalkam, Senthil Shen, Sheng Lee, Hannah Lo, Oscar Park, Jae Sung Guha, Etash Savarese, Silvio Schmidt, Ludwig Choi, Yejin Xiong, Caiming Xu, Ran |
| author_facet | Awadalla, Anas Xue, Le Shu, Manli Yan, An Wang, Jun Purushwalkam, Senthil Shen, Sheng Lee, Hannah Lo, Oscar Park, Jae Sung Guha, Etash Savarese, Silvio Schmidt, Ludwig Choi, Yejin Xiong, Caiming Xu, Ran |
| contents | We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text. KALE augments synthetic dense image captions with web-scale alt-text to generate factually grounded image captions. Our two-stage approach leverages large vision-language models and language models to create knowledge-augmented captions, which are then used to train a specialized VLM for scaling up the dataset. We train vision-language models on KALE and demonstrate improvements on vision-language tasks. Our experiments show the utility of KALE for training more capable and knowledgeable multimodal models. We release the KALE dataset at https://huggingface.co/datasets/Salesforce/blip3-kale |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07461 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions Awadalla, Anas Xue, Le Shu, Manli Yan, An Wang, Jun Purushwalkam, Senthil Shen, Sheng Lee, Hannah Lo, Oscar Park, Jae Sung Guha, Etash Savarese, Silvio Schmidt, Ludwig Choi, Yejin Xiong, Caiming Xu, Ran Computer Vision and Pattern Recognition Artificial Intelligence We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text. KALE augments synthetic dense image captions with web-scale alt-text to generate factually grounded image captions. Our two-stage approach leverages large vision-language models and language models to create knowledge-augmented captions, which are then used to train a specialized VLM for scaling up the dataset. We train vision-language models on KALE and demonstrate improvements on vision-language tasks. Our experiments show the utility of KALE for training more capable and knowledgeable multimodal models. We release the KALE dataset at https://huggingface.co/datasets/Salesforce/blip3-kale |
| title | BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2411.07461 |