Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.17686 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912711222231040 |
|---|---|
| author | Han, Yuhang Liu, Xuyang Zhang, Zihan Ding, Pengxiang Chen, Junjie Wang, Donglin Chen, Honggang Yan, Qingsen Huang, Siteng |
| author_facet | Han, Yuhang Liu, Xuyang Zhang, Zihan Ding, Pengxiang Chen, Junjie Wang, Donglin Chen, Honggang Yan, Qingsen Huang, Siteng |
| contents | The quadratic complexity of Multimodal Large Language Models (MLLMs) with respect to context length poses significant computational and memory challenges, hindering their real-world deployment. In the paper, we devise a ''filter-correlate-compress'' framework to accelerate the MLLM by systematically optimizing multimodal context length during prefilling. The framework first implements FiCoCo-V, a training-free method operating within the vision encoder. It employs a redundancy-based token discard mechanism that uses a novel integrated metric to accurately filter out redundant visual tokens. To mitigate information loss, the framework introduces a correlation-based information recycling mechanism that allows preserved tokens to selectively recycle information from correlated discarded tokens with a self-preserving compression, thereby preventing the dilution of their own core content. The framework's FiCoCo-L variant further leverages task-aware textual priors to perform token reduction directly within the LLM decoder. Extensive experiments demonstrate that the FiCoCo series effectively accelerates a range of MLLMs, achieves up to 14.7x FLOPs reduction with 93.6% performance retention. Our methods consistently outperform state-of-the-art training-free approaches, showcasing effectiveness and generalizability across model architectures, sizes, and tasks without requiring retraining. Code: https://github.com/kawhiiiileo/FiCoCo |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17686 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration Han, Yuhang Liu, Xuyang Zhang, Zihan Ding, Pengxiang Chen, Junjie Wang, Donglin Chen, Honggang Yan, Qingsen Huang, Siteng Computer Vision and Pattern Recognition The quadratic complexity of Multimodal Large Language Models (MLLMs) with respect to context length poses significant computational and memory challenges, hindering their real-world deployment. In the paper, we devise a ''filter-correlate-compress'' framework to accelerate the MLLM by systematically optimizing multimodal context length during prefilling. The framework first implements FiCoCo-V, a training-free method operating within the vision encoder. It employs a redundancy-based token discard mechanism that uses a novel integrated metric to accurately filter out redundant visual tokens. To mitigate information loss, the framework introduces a correlation-based information recycling mechanism that allows preserved tokens to selectively recycle information from correlated discarded tokens with a self-preserving compression, thereby preventing the dilution of their own core content. The framework's FiCoCo-L variant further leverages task-aware textual priors to perform token reduction directly within the LLM decoder. Extensive experiments demonstrate that the FiCoCo series effectively accelerates a range of MLLMs, achieves up to 14.7x FLOPs reduction with 93.6% performance retention. Our methods consistently outperform state-of-the-art training-free approaches, showcasing effectiveness and generalizability across model architectures, sizes, and tasks without requiring retraining. Code: https://github.com/kawhiiiileo/FiCoCo |
| title | Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.17686 |