VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914206252531712 |
|---|---|
| author | Qin, Shengling Yu, Hao Wu, Chenxin Li, Zheng Cao, Yizhong Zhuge, Zhengyang Zhou, Yuxin Yao, Wentao Zhang, Yi Wang, Zhengheng Bai, Shuai Zhang, Jianwei Lin, Junyang |
| author_facet | Qin, Shengling Yu, Hao Wu, Chenxin Li, Zheng Cao, Yizhong Zhuge, Zhengyang Zhou, Yuxin Yao, Wentao Zhang, Yi Wang, Zhengheng Bai, Shuai Zhang, Jianwei Lin, Junyang |
| contents | This paper presents VLCache, a cache reuse framework that exploits both Key-Value (KV) cache and encoder cache from prior multimodal inputs to eliminate costly recomputation when the same multimodal inputs recur. Unlike previous heuristic approaches, we formally identify the cumulative reuse error effect and demonstrate how to minimize the non-prefix cache reuse error effectively. We further analyze the varying importance of model layers and propose a dynamic, layer-aware recomputation strategy to balance accuracy and efficiency. Experimental results show that VLCache achieves an accuracy on par with full recomputation, while requiring only 2-5% of the tokens to compute, yielding 1.2x-16x TTFT speedups. We develop an experimental implementation of the proposed VLCache pipeline based on SGLang, enabling significantly faster inference in practical deployments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_12977 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inference Qin, Shengling Yu, Hao Wu, Chenxin Li, Zheng Cao, Yizhong Zhuge, Zhengyang Zhou, Yuxin Yao, Wentao Zhang, Yi Wang, Zhengheng Bai, Shuai Zhang, Jianwei Lin, Junyang Computer Vision and Pattern Recognition This paper presents VLCache, a cache reuse framework that exploits both Key-Value (KV) cache and encoder cache from prior multimodal inputs to eliminate costly recomputation when the same multimodal inputs recur. Unlike previous heuristic approaches, we formally identify the cumulative reuse error effect and demonstrate how to minimize the non-prefix cache reuse error effectively. We further analyze the varying importance of model layers and propose a dynamic, layer-aware recomputation strategy to balance accuracy and efficiency. Experimental results show that VLCache achieves an accuracy on par with full recomputation, while requiring only 2-5% of the tokens to compute, yielding 1.2x-16x TTFT speedups. We develop an experimental implementation of the proposed VLCache pipeline based on SGLang, enabling significantly faster inference in practical deployments. |
| title | VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inference |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.12977 |