Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909416490532864 |
|---|---|
| author | Deitke, Matt Clark, Christopher Lee, Sangho Tripathi, Rohun Yang, Yue Park, Jae Sung Salehi, Mohammadreza Muennighoff, Niklas Lo, Kyle Soldaini, Luca Lu, Jiasen Anderson, Taira Bransom, Erin Ehsani, Kiana Ngo, Huong Chen, YenSung Patel, Ajay Yatskar, Mark Callison-Burch, Chris Head, Andrew Hendrix, Rose Bastani, Favyen VanderBilt, Eli Lambert, Nathan Chou, Yvonne Chheda, Arnavi Sparks, Jenna Skjonsberg, Sam Schmitz, Michael Sarnat, Aaron Bischoff, Byron Walsh, Pete Newell, Chris Wolters, Piper Gupta, Tanmay Zeng, Kuo-Hao Borchardt, Jon Groeneveld, Dirk Nam, Crystal Lebrecht, Sophie Wittlif, Caitlin Schoenick, Carissa Michel, Oscar Krishna, Ranjay Weihs, Luca Smith, Noah A. Hajishirzi, Hannaneh Girshick, Ross Farhadi, Ali Kembhavi, Aniruddha |
| author_facet | Deitke, Matt Clark, Christopher Lee, Sangho Tripathi, Rohun Yang, Yue Park, Jae Sung Salehi, Mohammadreza Muennighoff, Niklas Lo, Kyle Soldaini, Luca Lu, Jiasen Anderson, Taira Bransom, Erin Ehsani, Kiana Ngo, Huong Chen, YenSung Patel, Ajay Yatskar, Mark Callison-Burch, Chris Head, Andrew Hendrix, Rose Bastani, Favyen VanderBilt, Eli Lambert, Nathan Chou, Yvonne Chheda, Arnavi Sparks, Jenna Skjonsberg, Sam Schmitz, Michael Sarnat, Aaron Bischoff, Byron Walsh, Pete Newell, Chris Wolters, Piper Gupta, Tanmay Zeng, Kuo-Hao Borchardt, Jon Groeneveld, Dirk Nam, Crystal Lebrecht, Sophie Wittlif, Caitlin Schoenick, Carissa Michel, Oscar Krishna, Ranjay Weihs, Luca Smith, Noah A. Hajishirzi, Hannaneh Girshick, Ross Farhadi, Ali Kembhavi, Aniruddha |
| contents | Today's most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets called PixMo, including a dataset of highly detailed image captions for pre-training, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code are available at https://molmo.allenai.org/blog. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17146 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models Deitke, Matt Clark, Christopher Lee, Sangho Tripathi, Rohun Yang, Yue Park, Jae Sung Salehi, Mohammadreza Muennighoff, Niklas Lo, Kyle Soldaini, Luca Lu, Jiasen Anderson, Taira Bransom, Erin Ehsani, Kiana Ngo, Huong Chen, YenSung Patel, Ajay Yatskar, Mark Callison-Burch, Chris Head, Andrew Hendrix, Rose Bastani, Favyen VanderBilt, Eli Lambert, Nathan Chou, Yvonne Chheda, Arnavi Sparks, Jenna Skjonsberg, Sam Schmitz, Michael Sarnat, Aaron Bischoff, Byron Walsh, Pete Newell, Chris Wolters, Piper Gupta, Tanmay Zeng, Kuo-Hao Borchardt, Jon Groeneveld, Dirk Nam, Crystal Lebrecht, Sophie Wittlif, Caitlin Schoenick, Carissa Michel, Oscar Krishna, Ranjay Weihs, Luca Smith, Noah A. Hajishirzi, Hannaneh Girshick, Ross Farhadi, Ali Kembhavi, Aniruddha Computer Vision and Pattern Recognition Computation and Language Machine Learning Today's most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets called PixMo, including a dataset of highly detailed image captions for pre-training, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code are available at https://molmo.allenai.org/blog. |
| title | Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2409.17146 |