AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918216016592896 |
|---|---|
| author | Zhang, Xinliang Zhu, Lei He, Hangzhou Zeng, Shuang Fu, Ourui Hu, Jiakui Yao, Zhengjian Lu, Yanye |
| author_facet | Zhang, Xinliang Zhu, Lei He, Hangzhou Zeng, Shuang Fu, Ourui Hu, Jiakui Yao, Zhengjian Lu, Yanye |
| contents | Multimodal Large Language Models (MLLMs) have demonstrated substantial value in unified text-image understanding and reasoning, primarily by converting images into sequences of patch-level tokens that align with their architectural paradigm. However, patch-level tokenization leads to a quadratic growth in image tokens, burdening MLLMs' understanding and reasoning with enormous computation and memory. Additionally, the traditional patch-wise scanning tokenization workflow misaligns with the human vision cognition system, further leading to hallucination and computational redundancy. To address this issue, we propose an object-level token merging strategy for Adaptive Token compression, revealing the consistency with human vision system. The experiments are conducted on multiple comprehensive benchmarks, which show that our approach averagely, utilizes only 10% tokens while achieving almost 96% of the vanilla model's performance. More extensive experimental results in comparison with relevant works demonstrate the superiority of our method in balancing compression ratio and performance. Our code will be available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_14169 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs Zhang, Xinliang Zhu, Lei He, Hangzhou Zeng, Shuang Fu, Ourui Hu, Jiakui Yao, Zhengjian Lu, Yanye Computer Vision and Pattern Recognition Artificial Intelligence Multimodal Large Language Models (MLLMs) have demonstrated substantial value in unified text-image understanding and reasoning, primarily by converting images into sequences of patch-level tokens that align with their architectural paradigm. However, patch-level tokenization leads to a quadratic growth in image tokens, burdening MLLMs' understanding and reasoning with enormous computation and memory. Additionally, the traditional patch-wise scanning tokenization workflow misaligns with the human vision cognition system, further leading to hallucination and computational redundancy. To address this issue, we propose an object-level token merging strategy for Adaptive Token compression, revealing the consistency with human vision system. The experiments are conducted on multiple comprehensive benchmarks, which show that our approach averagely, utilizes only 10% tokens while achieving almost 96% of the vanilla model's performance. More extensive experimental results in comparison with relevant works demonstrate the superiority of our method in balancing compression ratio and performance. Our code will be available. |
| title | AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2511.14169 |