Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.08966 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915238060752896 |
|---|---|
| author | Dhouib, Mohamed Buscaldi, Davide Vanier, Sonia Shabou, Aymen |
| author_facet | Dhouib, Mohamed Buscaldi, Davide Vanier, Sonia Shabou, Aymen |
| contents | Visual Language Models require substantial computational resources for inference due to the additional input tokens needed to represent visual information. However, these visual tokens often contain redundant and unimportant information, resulting in an unnecessarily high number of tokens. To address this, we introduce PACT, a method that reduces inference time and memory usage by pruning irrelevant tokens and merging visually redundant ones at an early layer of the language model. Our approach uses a novel importance metric to identify unimportant tokens without relying on attention scores, making it compatible with FlashAttention. We also propose a novel clustering algorithm, called Distance Bounded Density Peak Clustering, which efficiently clusters visual tokens while constraining the distances between elements within a cluster by a predefined threshold. We demonstrate the effectiveness of PACT through extensive experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08966 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models Dhouib, Mohamed Buscaldi, Davide Vanier, Sonia Shabou, Aymen Computer Vision and Pattern Recognition Visual Language Models require substantial computational resources for inference due to the additional input tokens needed to represent visual information. However, these visual tokens often contain redundant and unimportant information, resulting in an unnecessarily high number of tokens. To address this, we introduce PACT, a method that reduces inference time and memory usage by pruning irrelevant tokens and merging visually redundant ones at an early layer of the language model. Our approach uses a novel importance metric to identify unimportant tokens without relying on attention scores, making it compatible with FlashAttention. We also propose a novel clustering algorithm, called Distance Bounded Density Peak Clustering, which efficiently clusters visual tokens while constraining the distances between elements within a cluster by a predefined threshold. We demonstrate the effectiveness of PACT through extensive experiments. |
| title | PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.08966 |