Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917972961918976 |
|---|---|
| author | Lee, Jewon Song, Ki-Ung Yang, Seungmin Lim, Donguk Kim, Jaeyeon Shin, Wooksu Kim, Bo-Kyeong Lee, Yong Jae Kim, Tae-Ho |
| author_facet | Lee, Jewon Song, Ki-Ung Yang, Seungmin Lim, Donguk Kim, Jaeyeon Shin, Wooksu Kim, Bo-Kyeong Lee, Yong Jae Kim, Tae-Ho |
| contents | Visual token reduction lowers inference costs caused by extensive image features in large vision-language models (LVLMs). Unlike relevant studies that prune tokens in self-attention-only LVLMs, our work uniquely addresses cross-attention-based models, which achieve superior performance. We identify that the key-value (KV) cache size for image tokens in cross-attention layers significantly exceeds that of text tokens in self-attention layers, posing a major compute bottleneck. To mitigate this issue, we exploit the sparse nature in cross-attention maps to selectively prune redundant visual features. Our Trimmed Llama effectively reduces KV cache demands without requiring additional training. By benefiting from 50%-reduced visual features, our model can reduce inference latency and memory usage while achieving benchmark parity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_00557 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features Lee, Jewon Song, Ki-Ung Yang, Seungmin Lim, Donguk Kim, Jaeyeon Shin, Wooksu Kim, Bo-Kyeong Lee, Yong Jae Kim, Tae-Ho Computer Vision and Pattern Recognition Machine Learning Visual token reduction lowers inference costs caused by extensive image features in large vision-language models (LVLMs). Unlike relevant studies that prune tokens in self-attention-only LVLMs, our work uniquely addresses cross-attention-based models, which achieve superior performance. We identify that the key-value (KV) cache size for image tokens in cross-attention layers significantly exceeds that of text tokens in self-attention layers, posing a major compute bottleneck. To mitigate this issue, we exploit the sparse nature in cross-attention maps to selectively prune redundant visual features. Our Trimmed Llama effectively reduces KV cache demands without requiring additional training. By benefiting from 50%-reduced visual features, our model can reduce inference latency and memory usage while achieving benchmark parity. |
| title | Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2504.00557 |