Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.11095 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908959720341504 |
|---|---|
| author | Sun, Siyu Ren, Jing Liao, Zhaohe Mao, Dongxiao Ren, Xiangyuan Zhang, Yiyi Zhao, Haohua Lin, Weixiong Shaohua, Jiang Zhang, Liqing Zheng, Yuchao |
| author_facet | Sun, Siyu Ren, Jing Liao, Zhaohe Mao, Dongxiao Ren, Xiangyuan Zhang, Yiyi Zhao, Haohua Lin, Weixiong Shaohua, Jiang Zhang, Liqing Zheng, Yuchao |
| contents | Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_11095 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Bottleneck Tokens for Unified Multimodal Retrieval Sun, Siyu Ren, Jing Liao, Zhaohe Mao, Dongxiao Ren, Xiangyuan Zhang, Yiyi Zhao, Haohua Lin, Weixiong Shaohua, Jiang Zhang, Liqing Zheng, Yuchao Machine Learning Artificial Intelligence Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps. First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token (e.g., <EOS>) as the sequence-level representation, a mechanism never designed for information aggregation. Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it. We address both gaps with two complementary components. Architecturally, we introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism. For training, we propose Generative Information Condensation: a next-token prediction objective coupled with a Condensation Mask that severs the direct attention path from target tokens to query tokens. All predictive signals are thereby forced through the BToks, converting the generative loss into dense, token-level supervision for semantic compression. At inference time, only the input and BToks are processed in a single forward pass with negligible overhead over conventional last-token pooling. On MMEB-V2 (78 datasets, 3 modalities, 9 meta-tasks), our approach achieves state-of-the-art among 2B-scale methods under comparable data conditions, attaining an Overall score of 59.0 (+3.6 over VLM2Vec-V2) with substantial gains on semantically demanding tasks (e.g., +12.6 on Video-QA). |
| title | Bottleneck Tokens for Unified Multimodal Retrieval |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.11095 |