ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914772899856384 |
|---|---|
| author | Cai, Mu Liu, Haotian Park, Dennis Mustikovela, Siva Karthik Meyer, Gregory P. Chai, Yuning Lee, Yong Jae |
| author_facet | Cai, Mu Liu, Haotian Park, Dennis Mustikovela, Siva Karthik Meyer, Gregory P. Chai, Yuning Lee, Yong Jae |
| contents | While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a "red bounding box" or "pointed arrow". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_00784 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts Cai, Mu Liu, Haotian Park, Dennis Mustikovela, Siva Karthik Meyer, Gregory P. Chai, Yuning Lee, Yong Jae Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning While existing large vision-language multimodal models focus on whole image understanding, there is a prominent gap in achieving region-specific comprehension. Current approaches that use textual coordinates or spatial encodings often fail to provide a user-friendly interface for visual prompting. To address this challenge, we introduce a novel multimodal model capable of decoding arbitrary visual prompts. This allows users to intuitively mark images and interact with the model using natural cues like a "red bounding box" or "pointed arrow". Our simple design directly overlays visual markers onto the RGB image, eliminating the need for complex region encodings, yet achieves state-of-the-art performance on region-understanding tasks like Visual7W, PointQA, and Visual Commonsense Reasoning benchmark. Furthermore, we present ViP-Bench, a comprehensive benchmark to assess the capability of models in understanding visual prompts across multiple dimensions, enabling future research in this domain. Code, data, and model are publicly available. |
| title | ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2312.00784 |