SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914690726100992 |
|---|---|
| author | Wang, Zekun Chen, Jingchang Zhou, Wangchunshu Zhu, Haichao Liang, Jiafeng Shan, Liping Liu, Ming Xu, Dongliang Yang, Qing Qin, Bing |
| author_facet | Wang, Zekun Chen, Jingchang Zhou, Wangchunshu Zhu, Haichao Liang, Jiafeng Shan, Liping Liu, Ming Xu, Dongliang Yang, Qing Qin, Bing |
| contents | Despite achieving remarkable performance on various vision-language tasks, Transformer-based Vision-Language Models (VLMs) suffer from redundancy in inputs and parameters, significantly hampering their efficiency in real-world applications. Moreover, the degree of redundancy in token representations and model parameters, such as attention heads, varies significantly for different inputs. In light of the challenges, we propose SmartTrim, an adaptive acceleration framework for VLMs, which adjusts the computational overhead per instance. Specifically, we integrate lightweight modules into the original backbone to identify and prune redundant token representations and attention heads within each layer. Furthermore, we devise a self-distillation strategy to enhance the consistency between the predictions of the pruned model and its fully-capacity counterpart. Experimental results across various vision-language tasks consistently demonstrate that SmartTrim accelerates the original model by 2-3 times with minimal performance degradation, highlighting the effectiveness and efficiency compared to previous approaches. Code will be available at https://github.com/kugwzk/SmartTrim. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_15033 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models Wang, Zekun Chen, Jingchang Zhou, Wangchunshu Zhu, Haichao Liang, Jiafeng Shan, Liping Liu, Ming Xu, Dongliang Yang, Qing Qin, Bing Computation and Language Despite achieving remarkable performance on various vision-language tasks, Transformer-based Vision-Language Models (VLMs) suffer from redundancy in inputs and parameters, significantly hampering their efficiency in real-world applications. Moreover, the degree of redundancy in token representations and model parameters, such as attention heads, varies significantly for different inputs. In light of the challenges, we propose SmartTrim, an adaptive acceleration framework for VLMs, which adjusts the computational overhead per instance. Specifically, we integrate lightweight modules into the original backbone to identify and prune redundant token representations and attention heads within each layer. Furthermore, we devise a self-distillation strategy to enhance the consistency between the predictions of the pruned model and its fully-capacity counterpart. Experimental results across various vision-language tasks consistently demonstrate that SmartTrim accelerates the original model by 2-3 times with minimal performance degradation, highlighting the effectiveness and efficiency compared to previous approaches. Code will be available at https://github.com/kugwzk/SmartTrim. |
| title | SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2305.15033 |