GME: Improving Universal Multimodal Retrieval by Multimodal LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915220280049664 |
|---|---|
| author | Zhang, Xin Zhang, Yanzhao Xie, Wen Li, Mingxin Dai, Ziqi Long, Dingkun Xie, Pengjun Zhang, Meishan Li, Wenjie Zhang, Min |
| author_facet | Zhang, Xin Zhang, Yanzhao Xie, Wen Li, Mingxin Dai, Ziqi Long, Dingkun Xie, Pengjun Zhang, Meishan Li, Wenjie Zhang, Min |
| contents | Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model, where queries and candidates can consist of pure text, images, or a combination of both. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data. However, our preliminary experiments demonstrate that more diverse multimodal training data can further unlock the potential of MLLMs. Despite its effectiveness, the existing multimodal training data is highly imbalanced in terms of modality, which motivates us to develop a training data synthesis pipeline and construct a large-scale, high-quality fused-modal training dataset. Based on the synthetic training data, we develop the General Multimodal Embedder (GME), an MLLM-based dense retriever designed for UMR. Furthermore, we construct a comprehensive UMR Benchmark (UMRB) to evaluate the effectiveness of our approach. Experimental results show that our method achieves state-of-the-art performance among existing UMR methods. Last, we provide in-depth analyses of model scaling and training strategies, and perform ablation studies on both the model and synthetic data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_16855 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GME: Improving Universal Multimodal Retrieval by Multimodal LLMs Zhang, Xin Zhang, Yanzhao Xie, Wen Li, Mingxin Dai, Ziqi Long, Dingkun Xie, Pengjun Zhang, Meishan Li, Wenjie Zhang, Min Computation and Language Information Retrieval Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model, where queries and candidates can consist of pure text, images, or a combination of both. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data. However, our preliminary experiments demonstrate that more diverse multimodal training data can further unlock the potential of MLLMs. Despite its effectiveness, the existing multimodal training data is highly imbalanced in terms of modality, which motivates us to develop a training data synthesis pipeline and construct a large-scale, high-quality fused-modal training dataset. Based on the synthetic training data, we develop the General Multimodal Embedder (GME), an MLLM-based dense retriever designed for UMR. Furthermore, we construct a comprehensive UMR Benchmark (UMRB) to evaluate the effectiveness of our approach. Experimental results show that our method achieves state-of-the-art performance among existing UMR methods. Last, we provide in-depth analyses of model scaling and training strategies, and perform ablation studies on both the model and synthetic data. |
| title | GME: Improving Universal Multimodal Retrieval by Multimodal LLMs |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2412.16855 |