Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912420209885184 |
|---|---|
| author | Mitra, Chancharik Huang, Brandon Chai, Tianning Lin, Zhiqiu Arbelle, Assaf Feris, Rogerio Karlinsky, Leonid Darrell, Trevor Ramanan, Deva Herzig, Roei |
| author_facet | Mitra, Chancharik Huang, Brandon Chai, Tianning Lin, Zhiqiu Arbelle, Assaf Feris, Rogerio Karlinsky, Leonid Darrell, Trevor Ramanan, Deva Herzig, Roei |
| contents | Generative Large Multimodal Models (LMMs) like LLaVA and Qwen-VL excel at a wide variety of vision-language (VL) tasks. Despite strong performance, LMMs' generative outputs are not specialized for vision-language classification tasks (i.e., tasks with vision-language inputs and discrete labels) such as image classification and multiple-choice VQA. One key challenge in utilizing LMMs for these tasks is the extraction of useful features from generative LMMs. To overcome this, we propose an approach that leverages multimodal feature extraction from the LMM's latent space. Toward this end, we present Sparse Attention Vectors (SAVs) -- a finetuning-free method that leverages sparse attention head activations (fewer than 5% of the heads) in LMMs as strong feature representations. With only few-shot examples, SAVs demonstrate state-of-the-art performance compared to a variety of few-shot and finetuned baselines on a collection of vision-language classification tasks. Our experiments also imply that SAVs can scale in performance with additional examples and generalize to similar tasks, establishing SAVs as both effective and robust multimodal feature representations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_00142 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features Mitra, Chancharik Huang, Brandon Chai, Tianning Lin, Zhiqiu Arbelle, Assaf Feris, Rogerio Karlinsky, Leonid Darrell, Trevor Ramanan, Deva Herzig, Roei Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Generative Large Multimodal Models (LMMs) like LLaVA and Qwen-VL excel at a wide variety of vision-language (VL) tasks. Despite strong performance, LMMs' generative outputs are not specialized for vision-language classification tasks (i.e., tasks with vision-language inputs and discrete labels) such as image classification and multiple-choice VQA. One key challenge in utilizing LMMs for these tasks is the extraction of useful features from generative LMMs. To overcome this, we propose an approach that leverages multimodal feature extraction from the LMM's latent space. Toward this end, we present Sparse Attention Vectors (SAVs) -- a finetuning-free method that leverages sparse attention head activations (fewer than 5% of the heads) in LMMs as strong feature representations. With only few-shot examples, SAVs demonstrate state-of-the-art performance compared to a variety of few-shot and finetuned baselines on a collection of vision-language classification tasks. Our experiments also imply that SAVs can scale in performance with additional examples and generalize to similar tasks, establishing SAVs as both effective and robust multimodal feature representations. |
| title | Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2412.00142 |