Cover Image

MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Haonan, Liu, Hong, Luo, Yuping, Wang, Liang, Yang, Nan, Wei, Furu, Dou, Zhicheng
Format:	Preprint
Published:	2025
Subjects:	Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data
by: Chen, Haonan, et al.
Published: (2025)

Little Giants: Synthesizing High-Quality Embedding Data at Scale
by: Chen, Haonan, et al.
Published: (2024)

Chain-of-Retrieval Augmented Generation
by: Wang, Liang, et al.
Published: (2025)

Text Embeddings by Weakly-Supervised Contrastive Pre-training
by: Wang, Liang, et al.
Published: (2022)

Thinking Augmented Pre-training
by: Wang, Liang, et al.
Published: (2025)

Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors
by: Ma, Shengkun, et al.
Published: (2024)

e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings
by: Chen, Haonan, et al.
Published: (2026)

Improving Text Embeddings with Large Language Models
by: Wang, Liang, et al.
Published: (2023)

LongEmbed: Extending Embedding Models for Long Context Retrieval
by: Zhu, Dawei, et al.
Published: (2024)

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE
by: Chen, Junyi, et al.
Published: (2023)

Learning to Retrieve In-Context Examples for Large Language Models
by: Wang, Liang, et al.
Published: (2023)

Stable Language Model Pre-training by Reducing Embedding Variability
by: Chung, Woojin, et al.
Published: (2024)

Multilingual E5 Text Embeddings: A Technical Report
by: Wang, Liang, et al.
Published: (2024)

Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training
by: Luo, Gen, et al.
Published: (2024)

Revealing the Learning Dynamics of Long-Context Continual Pre-training
by: Liang, Yupu, et al.
Published: (2026)

SoftMCL: Soft Momentum Contrastive Learning for Fine-grained Sentiment-aware Pre-training
by: Wang, Jin, et al.
Published: (2024)

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
by: Zhu, Tong, et al.
Published: (2024)

MH-MoE: Multi-Head Mixture-of-Experts
by: Huang, Shaohan, et al.
Published: (2024)

Text-to-Code Generation with Modality-relative Pre-training
by: Christopoulou, Fenia, et al.
Published: (2024)

MELT: Materials-aware Continued Pre-training for Language Model Adaptation to Materials Science
by: Kim, Junho, et al.
Published: (2024)

Continual Pre-training of MoEs: How robust is your router?
by: Thérien, Benjamin, et al.
Published: (2025)

Query-oriented Data Augmentation for Session Search
by: Chen, Haonan, et al.
Published: (2024)

Scaling Agents via Continual Pre-training
by: Su, Liangcai, et al.
Published: (2025)

Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition
by: Shou, Yuntao, et al.
Published: (2023)

From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan
by: Yang, Lei, et al.
Published: (2025)

DeMPT: Decoding-enhanced Multi-phase Prompt Tuning for Making LLMs Be Better Context-aware Translators
by: Lyu, Xinglin, et al.
Published: (2024)

Towards Effective and Efficient Continual Pre-training of Large Language Models
by: Chen, Jie, et al.
Published: (2024)

On Initializing Transformers with Pre-trained Embeddings
by: Kim, Ha Young, et al.
Published: (2024)

Is Pre-training Truly Better Than Meta-Learning?
by: Miranda, Brando, et al.
Published: (2023)

Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning
by: Liang, Zhengyang, et al.
Published: (2024)

Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation
by: Chen, Haonan, et al.
Published: (2024)

Efficient Continual Pre-training by Mitigating the Stability Gap
by: Guo, Yiduo, et al.
Published: (2024)

Bag of Lies: Robustness in Continuous Pre-training BERT
by: Gevers, Ine, et al.
Published: (2024)

Generate, Not Recommend: Personalized Multimodal Content Generation
by: Liu, Jiongnan, et al.
Published: (2025)

Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training
by: Luo, Zheheng, et al.
Published: (2024)

Pre-trained Large Language Models for Financial Sentiment Analysis
by: Luo, Wei, et al.
Published: (2024)

Reinforcement Pre-Training
by: Dong, Qingxiu, et al.
Published: (2025)

Probing Language Models for Pre-training Data Detection
by: Liu, Zhenhua, et al.
Published: (2024)

DEPT: Decoupled Embeddings for Pre-training Language Models
by: Iacob, Alex, et al.
Published: (2024)

MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrieval
by: Wu, Xixi, et al.
Published: (2025)