xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909791462359040 |
|---|---|
| author | Xue, Le Shu, Manli Awadalla, Anas Wang, Jun Yan, An Purushwalkam, Senthil Zhou, Honglu Prabhu, Viraj Dai, Yutong Ryoo, Michael S Kendre, Shrikant Zhang, Jieyu Tseng, Shaoyen Lujan-Moreno, Gustavo A Olson, Matthew L Hinck, Musashi Cobbley, David Lal, Vasudev Qin, Can Zhang, Shu Chen, Chia-Chih Yu, Ning Tan, Juntao Awalgaonkar, Tulika Manoj Heinecke, Shelby Wang, Huan Choi, Yejin Schmidt, Ludwig Chen, Zeyuan Savarese, Silvio Niebles, Juan Carlos Xiong, Caiming Xu, Ran |
| author_facet | Xue, Le Shu, Manli Awadalla, Anas Wang, Jun Yan, An Purushwalkam, Senthil Zhou, Honglu Prabhu, Viraj Dai, Yutong Ryoo, Michael S Kendre, Shrikant Zhang, Jieyu Tseng, Shaoyen Lujan-Moreno, Gustavo A Olson, Matthew L Hinck, Musashi Cobbley, David Lal, Vasudev Qin, Can Zhang, Shu Chen, Chia-Chih Yu, Ning Tan, Juntao Awalgaonkar, Tulika Manoj Heinecke, Shelby Wang, Huan Choi, Yejin Schmidt, Ludwig Chen, Zeyuan Savarese, Silvio Niebles, Juan Carlos Xiong, Caiming Xu, Ran |
| contents | This paper introduces BLIP-3, an open framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. We release 4B and 14B models, including both the pre-trained base model and the instruction fine-tuned ones. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our models demonstrate competitive performance among open-source LMMs with similar model sizes. Our resulting LMMs demonstrate competitive performance among open-source LMMs with similar model sizes, with the ability to comprehend interleaved image-text inputs. Our training code, models, and all datasets used in this work, including the three largescale datasets we create and the preprocessed ones, will be open-sourced to better support the research community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_08872 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | xGen-MM (BLIP-3): A Family of Open Large Multimodal Models Xue, Le Shu, Manli Awadalla, Anas Wang, Jun Yan, An Purushwalkam, Senthil Zhou, Honglu Prabhu, Viraj Dai, Yutong Ryoo, Michael S Kendre, Shrikant Zhang, Jieyu Tseng, Shaoyen Lujan-Moreno, Gustavo A Olson, Matthew L Hinck, Musashi Cobbley, David Lal, Vasudev Qin, Can Zhang, Shu Chen, Chia-Chih Yu, Ning Tan, Juntao Awalgaonkar, Tulika Manoj Heinecke, Shelby Wang, Huan Choi, Yejin Schmidt, Ludwig Chen, Zeyuan Savarese, Silvio Niebles, Juan Carlos Xiong, Caiming Xu, Ran Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language This paper introduces BLIP-3, an open framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. We release 4B and 14B models, including both the pre-trained base model and the instruction fine-tuned ones. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our models demonstrate competitive performance among open-source LMMs with similar model sizes. Our resulting LMMs demonstrate competitive performance among open-source LMMs with similar model sizes, with the ability to comprehend interleaved image-text inputs. Our training code, models, and all datasets used in this work, including the three largescale datasets we create and the preprocessed ones, will be open-sourced to better support the research community. |
| title | xGen-MM (BLIP-3): A Family of Open Large Multimodal Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2408.08872 |