STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Guo, Yichen, Li, Hanze, Zhang, Zonghao, You, Jinhao, Tang, Kai, Huang, Xiande
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computer Vision and Pattern Recognition
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866918023990870016
author	Guo, Yichen Li, Hanze Zhang, Zonghao You, Jinhao Tang, Kai Huang, Xiande
author_facet	Guo, Yichen Li, Hanze Zhang, Zonghao You, Jinhao Tang, Kai Huang, Xiande
contents	Although large vision-language models (LVLMs) leverage rich visual token representations to achieve strong performance on multimodal tasks, these tokens also introduce significant computational overhead during inference. Existing training-free token pruning methods typically adopt a single-stage strategy, focusing either on visual self-attention or visual-textual cross-attention. However, such localized perspectives often overlook the broader information flow across the model, leading to substantial performance degradation, especially under high pruning ratios. In this work, we propose STAR (Stage-wise Attention-guided token Reduction), a training-free, plug-and-play framework that approaches token pruning from a global perspective. Instead of pruning at a single point, STAR performs attention-guided reduction in two complementary stages: an early-stage pruning based on visual self-attention to remove redundant low-level features, and a later-stage pruning guided by cross-modal attention to discard task-irrelevant tokens. This holistic approach allows STAR to significantly reduce computational cost while better preserving task-critical information. Extensive experiments across multiple LVLM architectures and benchmarks show that STAR achieves strong acceleration while maintaining comparable, and in some cases even improved performance.
format	Preprint
id	arxiv_https___arxiv_org_abs_2505_12359
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference Guo, Yichen Li, Hanze Zhang, Zonghao You, Jinhao Tang, Kai Huang, Xiande Machine Learning Computer Vision and Pattern Recognition Although large vision-language models (LVLMs) leverage rich visual token representations to achieve strong performance on multimodal tasks, these tokens also introduce significant computational overhead during inference. Existing training-free token pruning methods typically adopt a single-stage strategy, focusing either on visual self-attention or visual-textual cross-attention. However, such localized perspectives often overlook the broader information flow across the model, leading to substantial performance degradation, especially under high pruning ratios. In this work, we propose STAR (Stage-wise Attention-guided token Reduction), a training-free, plug-and-play framework that approaches token pruning from a global perspective. Instead of pruning at a single point, STAR performs attention-guided reduction in two complementary stages: an early-stage pruning based on visual self-attention to remove redundant low-level features, and a later-stage pruning guided by cross-modal attention to discard task-irrelevant tokens. This holistic approach allows STAR to significantly reduce computational cost while better preserving task-critical information. Extensive experiments across multiple LVLM architectures and benchmarks show that STAR achieves strong acceleration while maintaining comparable, and in some cases even improved performance.
title	STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference
topic	Machine Learning Computer Vision and Pattern Recognition
url	https://arxiv.org/abs/2505.12359