Visual Prompting in Multimodal Large Language Models: A Survey
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910617874464768 |
|---|---|
| author | Wu, Junda Zhang, Zhehao Xia, Yu Li, Xintong Xia, Zhaoyang Chang, Aaron Yu, Tong Kim, Sungchul Rossi, Ryan A. Zhang, Ruiyi Mitra, Subrata Metaxas, Dimitris N. Yao, Lina Shang, Jingbo McAuley, Julian |
| author_facet | Wu, Junda Zhang, Zhehao Xia, Yu Li, Xintong Xia, Zhaoyang Chang, Aaron Yu, Tong Kim, Sungchul Rossi, Ryan A. Zhang, Ruiyi Mitra, Subrata Metaxas, Dimitris N. Yao, Lina Shang, Jingbo McAuley, Julian |
| contents | Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompting methods that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, object referring, and compositional reasoning abilities. In addition, we provide a summary of model training and in-context learning methods to improve MLLM's perception and understanding of visual prompts. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15310 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Visual Prompting in Multimodal Large Language Models: A Survey Wu, Junda Zhang, Zhehao Xia, Yu Li, Xintong Xia, Zhaoyang Chang, Aaron Yu, Tong Kim, Sungchul Rossi, Ryan A. Zhang, Ruiyi Mitra, Subrata Metaxas, Dimitris N. Yao, Lina Shang, Jingbo McAuley, Julian Machine Learning Computer Vision and Pattern Recognition Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompting methods that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, object referring, and compositional reasoning abilities. In addition, we provide a summary of model training and in-context learning methods to improve MLLM's perception and understanding of visual prompts. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods. |
| title | Visual Prompting in Multimodal Large Language Models: A Survey |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.15310 |