Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.16150 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915628441403392 |
|---|---|
| author | Liu, Chunxu Yang, Jiyuan Gao, Ruopeng Zhu, Yuhan Zhu, Feng Zhao, Rui Wang, Limin |
| author_facet | Liu, Chunxu Yang, Jiyuan Gao, Ruopeng Zhu, Yuhan Zhu, Feng Zhao, Rui Wang, Limin |
| contents | Multimodal embeddings are widely used in downstream tasks such as multimodal retrieval, enabling alignment of interleaved modalities in a shared representation space. While recent studies show that Multimodal Large Language Models (MLLMs) can serve as strong embedding extractors, existing approaches treat embedding extraction as a direct encoding step, overlooking the fact that MLLMs possess the generative capability for reasoning that could be leveraged to enhance representation quality. In this work, we explore how to explicitly incorporate reasoning into the embedding process. To this end, we propose Reasoning Guided Embeddings (RGE), which preserves the generative rationale process of MLLMs and couples it with contrastive training. Our method first enables the model to perform structured rationale generation conditioned on the instruction, and then extracts representations after reasoning has unfolded. This simple design enhances the context-conditional inference signals within the embedding, leading to improved multimodal representation quality. Experiments on the MMEB benchmark show that reasoning-guided conditioning improves multimodal retrieval performance by 4.9% over the non-reasoning baseline, confirming that explicit reasoning can effectively enhance embedding quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_16150 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval Liu, Chunxu Yang, Jiyuan Gao, Ruopeng Zhu, Yuhan Zhu, Feng Zhao, Rui Wang, Limin Computer Vision and Pattern Recognition Multimodal embeddings are widely used in downstream tasks such as multimodal retrieval, enabling alignment of interleaved modalities in a shared representation space. While recent studies show that Multimodal Large Language Models (MLLMs) can serve as strong embedding extractors, existing approaches treat embedding extraction as a direct encoding step, overlooking the fact that MLLMs possess the generative capability for reasoning that could be leveraged to enhance representation quality. In this work, we explore how to explicitly incorporate reasoning into the embedding process. To this end, we propose Reasoning Guided Embeddings (RGE), which preserves the generative rationale process of MLLMs and couples it with contrastive training. Our method first enables the model to perform structured rationale generation conditioned on the instruction, and then extracts representations after reasoning has unfolded. This simple design enhances the context-conditional inference signals within the embedding, leading to improved multimodal representation quality. Experiments on the MMEB benchmark show that reasoning-guided conditioning improves multimodal retrieval performance by 4.9% over the non-reasoning baseline, confirming that explicit reasoning can effectively enhance embedding quality. |
| title | Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.16150 |