Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2602.19091 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918349705838592 |
|---|---|
| author | Liu, Lihao Wang, Yan Yang, Biao Li, Da Cao, Jiangxia Luo, Yuxiao Chen, Xiang Wu, Xiangyu Yuan, Wei Yang, Fan Ding, Guiguang Gao, Tingting Zhou, Guorui |
| author_facet | Liu, Lihao Wang, Yan Yang, Biao Li, Da Cao, Jiangxia Luo, Yuxiao Chen, Xiang Wu, Xiangyu Yuan, Wei Yang, Fan Ding, Guiguang Gao, Tingting Zhou, Guorui |
| contents | Multimodal Large Language Models (MLLMs) have shown remarkable success in comprehension tasks such as visual description and visual question answering. However, their direct application to embedding-based tasks like retrieval remains challenging due to the discrepancy between output formats and optimization objectives. Previous approaches often employ contrastive fine-tuning to adapt MLLMs for retrieval, but at the cost of losing their generative capabilities. We argue that both generative and embedding tasks fundamentally rely on shared cognitive mechanisms, specifically cross-modal representation alignment and contextual comprehension. To this end, we propose CREM (Compression-driven Representation Enhanced Model), with a unified framework that enhances multimodal representations for retrieval while preserving generative ability. Specifically, we introduce a compression-based prompt design with learnable chorus tokens to aggregate multimodal semantics and a compression-driven training strategy that integrates contrastive and generative objectives through compression-aware attention. Extensive experiments demonstrate that CREM achieves state-of-the-art retrieval performance on MMEB while maintaining strong generative performance on multiple comprehension benchmarks. Our findings highlight that generative supervision can further improve the representational quality of MLLMs under the proposed compression-driven paradigm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_19091 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension Liu, Lihao Wang, Yan Yang, Biao Li, Da Cao, Jiangxia Luo, Yuxiao Chen, Xiang Wu, Xiangyu Yuan, Wei Yang, Fan Ding, Guiguang Gao, Tingting Zhou, Guorui Computer Vision and Pattern Recognition Multimodal Large Language Models (MLLMs) have shown remarkable success in comprehension tasks such as visual description and visual question answering. However, their direct application to embedding-based tasks like retrieval remains challenging due to the discrepancy between output formats and optimization objectives. Previous approaches often employ contrastive fine-tuning to adapt MLLMs for retrieval, but at the cost of losing their generative capabilities. We argue that both generative and embedding tasks fundamentally rely on shared cognitive mechanisms, specifically cross-modal representation alignment and contextual comprehension. To this end, we propose CREM (Compression-driven Representation Enhanced Model), with a unified framework that enhances multimodal representations for retrieval while preserving generative ability. Specifically, we introduce a compression-based prompt design with learnable chorus tokens to aggregate multimodal semantics and a compression-driven training strategy that integrates contrastive and generative objectives through compression-aware attention. Extensive experiments demonstrate that CREM achieves state-of-the-art retrieval performance on MMEB while maintaining strong generative performance on multiple comprehension benchmarks. Our findings highlight that generative supervision can further improve the representational quality of MLLMs under the proposed compression-driven paradigm. |
| title | CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.19091 |