ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929661218390016 |
|---|---|
| author | Wu, Mingrui Cai, Xinyue Ji, Jiayi Li, Jiale Huang, Oucheng Luo, Gen Fei, Hao Jiang, Guannan Sun, Xiaoshuai Ji, Rongrong |
| author_facet | Wu, Mingrui Cai, Xinyue Ji, Jiayi Li, Jiale Huang, Oucheng Luo, Gen Fei, Hao Jiang, Guannan Sun, Xiaoshuai Ji, Rongrong |
| contents | In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through test-time optimization of a learnable latent variable. We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MLP output at test time, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point. The results demonstrate that our method exhibits out-of-domain generalization and interpretability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_21534 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models Wu, Mingrui Cai, Xinyue Ji, Jiayi Li, Jiale Huang, Oucheng Luo, Gen Fei, Hao Jiang, Guannan Sun, Xiaoshuai Ji, Rongrong Computer Vision and Pattern Recognition In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through test-time optimization of a learnable latent variable. We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MLP output at test time, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point. The results demonstrate that our method exhibits out-of-domain generalization and interpretability. |
| title | ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.21534 |