EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917601313030144 |
|---|---|
| author | Chen, Junyi Guo, Longteng Sun, Jia Shao, Shuai Yuan, Zehuan Lin, Liang Zhang, Dongyu |
| author_facet | Chen, Junyi Guo, Longteng Sun, Jia Shao, Shuai Yuan, Zehuan Lin, Liang Zhang, Dongyu |
| contents | Building scalable vision-language models to learn from diverse, multimodal data remains an open challenge. In this paper, we introduce an Efficient Vision-languagE foundation model, namely EVE, which is one unified multimodal Transformer pre-trained solely by one unified pre-training task. Specifically, EVE encodes both vision and language within a shared Transformer network integrated with modality-aware sparse Mixture-of-Experts (MoE) modules, which capture modality-specific information by selectively switching to different experts. To unify pre-training tasks of vision and language, EVE performs masked signal modeling on image-text pairs to reconstruct masked signals, i.e., image pixels and text tokens, given visible signals. This simple yet effective pre-training objective accelerates training by 3.5x compared to the model pre-trained with Image-Text Contrastive and Image-Text Matching losses. Owing to the combination of the unified architecture and pre-training task, EVE is easy to scale up, enabling better downstream performance with fewer resources and faster training speed. Despite its simplicity, EVE achieves state-of-the-art performance on various vision-language downstream tasks, including visual question answering, visual reasoning, and image-text retrieval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_11971 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE Chen, Junyi Guo, Longteng Sun, Jia Shao, Shuai Yuan, Zehuan Lin, Liang Zhang, Dongyu Computer Vision and Pattern Recognition Computation and Language Machine Learning Multimedia Building scalable vision-language models to learn from diverse, multimodal data remains an open challenge. In this paper, we introduce an Efficient Vision-languagE foundation model, namely EVE, which is one unified multimodal Transformer pre-trained solely by one unified pre-training task. Specifically, EVE encodes both vision and language within a shared Transformer network integrated with modality-aware sparse Mixture-of-Experts (MoE) modules, which capture modality-specific information by selectively switching to different experts. To unify pre-training tasks of vision and language, EVE performs masked signal modeling on image-text pairs to reconstruct masked signals, i.e., image pixels and text tokens, given visible signals. This simple yet effective pre-training objective accelerates training by 3.5x compared to the model pre-trained with Image-Text Contrastive and Image-Text Matching losses. Owing to the combination of the unified architecture and pre-training task, EVE is easy to scale up, enabling better downstream performance with fewer resources and faster training speed. Despite its simplicity, EVE achieves state-of-the-art performance on various vision-language downstream tasks, including visual question answering, visual reasoning, and image-text retrieval. |
| title | EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning Multimedia |
| url | https://arxiv.org/abs/2308.11971 |