ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909883727609856 |
|---|---|
| author | Yuan, Qianhao Zhang, Qingyu Liu, Yanjiang Chen, Jiawei Lu, Yaojie Lin, Hongyu Zheng, Jia Han, Xianpei Sun, Le |
| author_facet | Yuan, Qianhao Zhang, Qingyu Liu, Yanjiang Chen, Jiawei Lu, Yaojie Lin, Hongyu Zheng, Jia Han, Xianpei Sun, Le |
| contents | Multimodal Large Language Models (MLLMs) suffer from high computational costs due to their massive size and the large number of visual tokens. In this paper, we investigate layer-wise redundancy in MLLMs by introducing a novel metric, Layer Contribution (LC), which quantifies the impact of a layer's transformations on visual and text tokens, respectively. The calculation of LC involves measuring the divergence in model output that results from removing the layer's transformations on the specified tokens. Our pilot experiment reveals that many layers of MLLMs exhibit minimal contribution during the processing of visual tokens. Motivated by this observation, we propose ShortV, a training-free method that leverages LC to identify ineffective layers, and freezes visual token updates in these layers. Experiments show that ShortV can freeze visual token in approximately 60\% of the MLLM layers, thereby dramatically reducing computational costs related to updating visual tokens. For example, it achieves a 50\% reduction in FLOPs on LLaVA-NeXT-13B while maintaining superior performance. The code will be publicly available at https://github.com/icip-cas/ShortV |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_00502 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers Yuan, Qianhao Zhang, Qingyu Liu, Yanjiang Chen, Jiawei Lu, Yaojie Lin, Hongyu Zheng, Jia Han, Xianpei Sun, Le Computer Vision and Pattern Recognition Computation and Language Multimodal Large Language Models (MLLMs) suffer from high computational costs due to their massive size and the large number of visual tokens. In this paper, we investigate layer-wise redundancy in MLLMs by introducing a novel metric, Layer Contribution (LC), which quantifies the impact of a layer's transformations on visual and text tokens, respectively. The calculation of LC involves measuring the divergence in model output that results from removing the layer's transformations on the specified tokens. Our pilot experiment reveals that many layers of MLLMs exhibit minimal contribution during the processing of visual tokens. Motivated by this observation, we propose ShortV, a training-free method that leverages LC to identify ineffective layers, and freezes visual token updates in these layers. Experiments show that ShortV can freeze visual token in approximately 60\% of the MLLM layers, thereby dramatically reducing computational costs related to updating visual tokens. For example, it achieves a 50\% reduction in FLOPs on LLaVA-NeXT-13B while maintaining superior performance. The code will be publicly available at https://github.com/icip-cas/ShortV |
| title | ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2504.00502 |