Think Then Embed: Generative Context Improves Multimodal Embedding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911385858867200 |
|---|---|
| author | Cui, Xuanming Cheng, Jianpeng Chen, Hong-you Shukla, Satya Narayan Awasthi, Abhijeet Pan, Xichen Ahuja, Chaitanya Mishra, Shlok Kumar Yang, Yonghuan Xiao, Jun Guo, Qi Lim, Ser-Nam Singh, Aashu Fan, Xiangjun |
| author_facet | Cui, Xuanming Cheng, Jianpeng Chen, Hong-you Shukla, Satya Narayan Awasthi, Abhijeet Pan, Xichen Ahuja, Chaitanya Mishra, Shlok Kumar Yang, Yonghuan Xiao, Jun Guo, Qi Lim, Ser-Nam Singh, Aashu Fan, Xiangjun |
| contents | There is a growing interest in Universal Multimodal Embeddings (UME), where models are required to generate task-specific representations. While recent studies show that Multimodal Large Language Models (MLLMs) perform well on such tasks, they treat MLLMs solely as encoders, overlooking their generative capacity. However, such an encoding paradigm becomes less effective as instructions become more complex and require compositional reasoning. Inspired by the proven effectiveness of chain-of-thought reasoning, we propose a general Think-Then-Embed (TTE) framework for UME, composed of a reasoner and an embedder. The reasoner MLLM first generates reasoning traces that explain complex queries, followed by an embedder that produces representations conditioned on both the original query and the intermediate reasoning. This explicit reasoning step enables more nuanced understanding of complex multimodal instructions. Our contributions are threefold. First, by leveraging a powerful MLLM reasoner, we achieve state-of-the-art performance on the MMEB-V2 benchmark, surpassing proprietary models trained on massive in-house datasets. Second, to reduce the dependency on large MLLM reasoners, we finetune a smaller MLLM reasoner using high-quality embedding-centric reasoning traces, achieving the best performance among open-source models with a 7% absolute gain over recently proposed models. Third, we investigate strategies for integrating the reasoner and embedder into a unified model for improved efficiency without sacrificing performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_05014 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Think Then Embed: Generative Context Improves Multimodal Embedding Cui, Xuanming Cheng, Jianpeng Chen, Hong-you Shukla, Satya Narayan Awasthi, Abhijeet Pan, Xichen Ahuja, Chaitanya Mishra, Shlok Kumar Yang, Yonghuan Xiao, Jun Guo, Qi Lim, Ser-Nam Singh, Aashu Fan, Xiangjun Artificial Intelligence Machine Learning There is a growing interest in Universal Multimodal Embeddings (UME), where models are required to generate task-specific representations. While recent studies show that Multimodal Large Language Models (MLLMs) perform well on such tasks, they treat MLLMs solely as encoders, overlooking their generative capacity. However, such an encoding paradigm becomes less effective as instructions become more complex and require compositional reasoning. Inspired by the proven effectiveness of chain-of-thought reasoning, we propose a general Think-Then-Embed (TTE) framework for UME, composed of a reasoner and an embedder. The reasoner MLLM first generates reasoning traces that explain complex queries, followed by an embedder that produces representations conditioned on both the original query and the intermediate reasoning. This explicit reasoning step enables more nuanced understanding of complex multimodal instructions. Our contributions are threefold. First, by leveraging a powerful MLLM reasoner, we achieve state-of-the-art performance on the MMEB-V2 benchmark, surpassing proprietary models trained on massive in-house datasets. Second, to reduce the dependency on large MLLM reasoners, we finetune a smaller MLLM reasoner using high-quality embedding-centric reasoning traces, achieving the best performance among open-source models with a 7% absolute gain over recently proposed models. Third, we investigate strategies for integrating the reasoner and embedder into a unified model for improved efficiency without sacrificing performance. |
| title | Think Then Embed: Generative Context Improves Multimodal Embedding |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2510.05014 |