Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.09927 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913690808221696 |
|---|---|
| author | Granite Vision Team Karlinsky, Leonid Arbelle, Assaf Daniels, Abraham Nassar, Ahmed Alfassi, Amit Wu, Bo Schwartz, Eli Joshi, Dhiraj Kondic, Jovana Shabtay, Nimrod Li, Pengyuan Herzig, Roei Abedin, Shafiq Perek, Shaked Harary, Sivan Barzelay, Udi Goldfarb, Adi Raz Oliva, Aude Wieles, Ben Bhattacharjee, Bishwaranjan Huang, Brandon Auer, Christoph Gutfreund, Dan Beymer, David Wood, David Kuehne, Hilde Hansen, Jacob Shtok, Joseph Wong, Ken Bathen, Luis Angel Mishra, Mayank Lysak, Maksym Dolfi, Michele Yurochkin, Mikhail Livathinos, Nikolaos Harel, Nimrod Azulai, Ophir Naparstek, Oshri de Lima, Rafael Teixeira Panda, Rameswar Doveh, Sivan Gupta, Shubham Das, Subhro Zawad, Syed Kim, Yusik He, Zexue Brooks, Alexander Goodhart, Gabe Govindjee, Anita Leist, Derek Ibrahim, Ibrahim Soffer, Aya Cox, David Soule, Kate Lastras, Luis Desai, Nirmit Ofek-koifman, Shila Raghavan, Sriram Syeda-Mahmood, Tanveer Staar, Peter Drory, Tal Feris, Rogerio |
| author_facet | Granite Vision Team Karlinsky, Leonid Arbelle, Assaf Daniels, Abraham Nassar, Ahmed Alfassi, Amit Wu, Bo Schwartz, Eli Joshi, Dhiraj Kondic, Jovana Shabtay, Nimrod Li, Pengyuan Herzig, Roei Abedin, Shafiq Perek, Shaked Harary, Sivan Barzelay, Udi Goldfarb, Adi Raz Oliva, Aude Wieles, Ben Bhattacharjee, Bishwaranjan Huang, Brandon Auer, Christoph Gutfreund, Dan Beymer, David Wood, David Kuehne, Hilde Hansen, Jacob Shtok, Joseph Wong, Ken Bathen, Luis Angel Mishra, Mayank Lysak, Maksym Dolfi, Michele Yurochkin, Mikhail Livathinos, Nikolaos Harel, Nimrod Azulai, Ophir Naparstek, Oshri de Lima, Rafael Teixeira Panda, Rameswar Doveh, Sivan Gupta, Shubham Das, Subhro Zawad, Syed Kim, Yusik He, Zexue Brooks, Alexander Goodhart, Gabe Govindjee, Anita Leist, Derek Ibrahim, Ibrahim Soffer, Aya Cox, David Soule, Kate Lastras, Luis Desai, Nirmit Ofek-koifman, Shila Raghavan, Sriram Syeda-Mahmood, Tanveer Staar, Peter Drory, Tal Feris, Rogerio |
| contents | We introduce Granite Vision, a lightweight large language model with vision capabilities, specifically designed to excel in enterprise use cases, particularly in visual document understanding. Our model is trained on a comprehensive instruction-following dataset, including document-related tasks, such as content extraction from tables, charts, diagrams, sketches, and infographics, as well as general image tasks. The architecture of Granite Vision is centered around visual modality alignment with a decoder-only, 2 billion parameter Granite large language model. Additionally, we introduce a dedicated safety classification approach in test-time that leverages a sparse set of attention vectors to identify potential harmful inputs. Despite its lightweight architecture, Granite Vision achieves strong results in standard benchmarks related to visual document understanding, as well as on the LiveXiv benchmark, which is designed to avoid test set contamination by using a constantly updated corpus of recently published Arxiv papers. We are releasing the model under the Apache-2 license, allowing for both research and commercial use, while offering complete visibility into the training data and other relevant details. See https://huggingface.co/ibm-granite/ for model weights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_09927 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence Granite Vision Team Karlinsky, Leonid Arbelle, Assaf Daniels, Abraham Nassar, Ahmed Alfassi, Amit Wu, Bo Schwartz, Eli Joshi, Dhiraj Kondic, Jovana Shabtay, Nimrod Li, Pengyuan Herzig, Roei Abedin, Shafiq Perek, Shaked Harary, Sivan Barzelay, Udi Goldfarb, Adi Raz Oliva, Aude Wieles, Ben Bhattacharjee, Bishwaranjan Huang, Brandon Auer, Christoph Gutfreund, Dan Beymer, David Wood, David Kuehne, Hilde Hansen, Jacob Shtok, Joseph Wong, Ken Bathen, Luis Angel Mishra, Mayank Lysak, Maksym Dolfi, Michele Yurochkin, Mikhail Livathinos, Nikolaos Harel, Nimrod Azulai, Ophir Naparstek, Oshri de Lima, Rafael Teixeira Panda, Rameswar Doveh, Sivan Gupta, Shubham Das, Subhro Zawad, Syed Kim, Yusik He, Zexue Brooks, Alexander Goodhart, Gabe Govindjee, Anita Leist, Derek Ibrahim, Ibrahim Soffer, Aya Cox, David Soule, Kate Lastras, Luis Desai, Nirmit Ofek-koifman, Shila Raghavan, Sriram Syeda-Mahmood, Tanveer Staar, Peter Drory, Tal Feris, Rogerio Computer Vision and Pattern Recognition Artificial Intelligence We introduce Granite Vision, a lightweight large language model with vision capabilities, specifically designed to excel in enterprise use cases, particularly in visual document understanding. Our model is trained on a comprehensive instruction-following dataset, including document-related tasks, such as content extraction from tables, charts, diagrams, sketches, and infographics, as well as general image tasks. The architecture of Granite Vision is centered around visual modality alignment with a decoder-only, 2 billion parameter Granite large language model. Additionally, we introduce a dedicated safety classification approach in test-time that leverages a sparse set of attention vectors to identify potential harmful inputs. Despite its lightweight architecture, Granite Vision achieves strong results in standard benchmarks related to visual document understanding, as well as on the LiveXiv benchmark, which is designed to avoid test set contamination by using a constantly updated corpus of recently published Arxiv papers. We are releasing the model under the Apache-2 license, allowing for both research and commercial use, while offering complete visibility into the training data and other relevant details. See https://huggingface.co/ibm-granite/ for model weights. |
| title | Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2502.09927 |