Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.19333 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929364490256384 |
|---|---|
| author | Ma, Feipeng Xue, Hongwei Wang, Guangting Zhou, Yizhou Rao, Fengyun Yan, Shilin Zhang, Yueyi Wu, Siying Shou, Mike Zheng Sun, Xiaoyan |
| author_facet | Ma, Feipeng Xue, Hongwei Wang, Guangting Zhou, Yizhou Rao, Fengyun Yan, Shilin Zhang, Yueyi Wu, Siying Shou, Mike Zheng Sun, Xiaoyan |
| contents | Most multi-modal tasks can be formulated into problems of either generation or embedding. Existing models usually tackle these two types of problems by decoupling language modules into a text decoder for generation, and a text encoder for embedding. To explore the minimalism of multi-modal paradigms, we attempt to achieve only one model per modality in this work. We propose a Multi-Modal Generative Embedding Model (MM-GEM), whereby the generative and embedding objectives are encapsulated in one Large Language Model. We also propose a PoolAggregator to boost efficiency and enable the ability of fine-grained embedding and generation. A surprising finding is that these two objectives do not significantly conflict with each other. For example, MM-GEM instantiated from ViT-Large and TinyLlama shows competitive performance on benchmarks for multimodal embedding models such as cross-modal retrieval and zero-shot classification, while has good ability of image captioning. Additionally, MM-GEM can seamlessly execute region-level image caption generation and retrieval tasks. Besides, the advanced text model in MM-GEM brings over 5% improvement in Recall@1 for long text and image retrieval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_19333 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-Modal Generative Embedding Model Ma, Feipeng Xue, Hongwei Wang, Guangting Zhou, Yizhou Rao, Fengyun Yan, Shilin Zhang, Yueyi Wu, Siying Shou, Mike Zheng Sun, Xiaoyan Computer Vision and Pattern Recognition Most multi-modal tasks can be formulated into problems of either generation or embedding. Existing models usually tackle these two types of problems by decoupling language modules into a text decoder for generation, and a text encoder for embedding. To explore the minimalism of multi-modal paradigms, we attempt to achieve only one model per modality in this work. We propose a Multi-Modal Generative Embedding Model (MM-GEM), whereby the generative and embedding objectives are encapsulated in one Large Language Model. We also propose a PoolAggregator to boost efficiency and enable the ability of fine-grained embedding and generation. A surprising finding is that these two objectives do not significantly conflict with each other. For example, MM-GEM instantiated from ViT-Large and TinyLlama shows competitive performance on benchmarks for multimodal embedding models such as cross-modal retrieval and zero-shot classification, while has good ability of image captioning. Additionally, MM-GEM can seamlessly execute region-level image caption generation and retrieval tasks. Besides, the advanced text model in MM-GEM brings over 5% improvement in Recall@1 for long text and image retrieval. |
| title | Multi-Modal Generative Embedding Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.19333 |