Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915944013496320 |
|---|---|
| author | Liao, Chenfei Wang, Wensong Wen, Zichen Zheng, Xu Wang, Yiyu He, Haocong Lyu, Yuanhuiyi Jiang, Lutao Zou, Xin Fu, Yuqian Ren, Bin Zhang, Linfeng Hu, Xuming |
| author_facet | Liao, Chenfei Wang, Wensong Wen, Zichen Zheng, Xu Wang, Yiyu He, Haocong Lyu, Yuanhuiyi Jiang, Lutao Zou, Xin Fu, Yuqian Ren, Bin Zhang, Linfeng Hu, Xuming |
| contents | Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression. The effectiveness of these methods is commonly evaluated by measuring the accuracy drop on existing MLLM benchmarks before and after compression. However, these benchmarks are originally designed to assess general perception and reasoning abilities, rather than the specific challenges posed by visual token compression, leading to a fundamental task mismatch. In this work, we uncover a counterintuitive yet consistent phenomenon: simple image downsampling outperforms many advanced visual token compression methods across multiple widely used benchmarks. Through a comprehensive empirical study spanning eight popular benchmarks and multiple state-of-the-art compression techniques, we show that (i) current benchmarks contain substantial noise (task-irrelevant samples) for evaluating visual token compression, and (ii) downsampling can act as an effective data filter that distinguishes between simple and difficult samples with respect to compression sensitivity. Motivated by these findings, we propose VTC-Bench, an evaluation framework that explicitly leverages downsampling as a discriminator to denoise existing benchmarks, enabling a fairer and more meaningful additional assessment of visual token compression methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07143 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods Liao, Chenfei Wang, Wensong Wen, Zichen Zheng, Xu Wang, Yiyu He, Haocong Lyu, Yuanhuiyi Jiang, Lutao Zou, Xin Fu, Yuqian Ren, Bin Zhang, Linfeng Hu, Xuming Computer Vision and Pattern Recognition Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression. The effectiveness of these methods is commonly evaluated by measuring the accuracy drop on existing MLLM benchmarks before and after compression. However, these benchmarks are originally designed to assess general perception and reasoning abilities, rather than the specific challenges posed by visual token compression, leading to a fundamental task mismatch. In this work, we uncover a counterintuitive yet consistent phenomenon: simple image downsampling outperforms many advanced visual token compression methods across multiple widely used benchmarks. Through a comprehensive empirical study spanning eight popular benchmarks and multiple state-of-the-art compression techniques, we show that (i) current benchmarks contain substantial noise (task-irrelevant samples) for evaluating visual token compression, and (ii) downsampling can act as an effective data filter that distinguishes between simple and difficult samples with respect to compression sensitivity. Motivated by these findings, we propose VTC-Bench, an evaluation framework that explicitly leverages downsampling as a discriminator to denoise existing benchmarks, enabling a fairer and more meaningful additional assessment of visual token compression methods. |
| title | Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.07143 |