MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908599517708288 |
|---|---|
| author | Wu, Qiyu Cui, Shuyang Hayakawa, Satoshi Wang, Wei-Yao Wakaki, Hiromi Mitsufuji, Yuki |
| author_facet | Wu, Qiyu Cui, Shuyang Hayakawa, Satoshi Wang, Wei-Yao Wakaki, Hiromi Mitsufuji, Yuki |
| contents | Multimodal retrieval, which seeks to retrieve relevant content across modalities such as text or image, supports applications from AI search to contents production. Despite the success of separate-encoder approaches like CLIP align modality-specific embeddings with contrastive learning, recent multimodal large language models (MLLMs) enable a unified encoder that directly processes composed inputs. While flexible and advanced, we identify that unified encoders trained with conventional contrastive learning are prone to learn modality shortcut, leading to poor robustness under distribution shifts. We propose a modality composition awareness framework to mitigate this issue. Concretely, a preference loss enforces multimodal embeddings to outperform their unimodal counterparts, while a composition regularization objective aligns multimodal embeddings with prototypes composed from its unimodal parts. These objectives explicitly model structural relationships between the composed representation and its unimodal counterparts. Experiments on various benchmarks show gains in out-of-distribution retrieval, highlighting modality composition awareness as a effective principle for robust composed multimodal retrieval when utilizing MLLMs as the unified encoder. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_15543 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval Wu, Qiyu Cui, Shuyang Hayakawa, Satoshi Wang, Wei-Yao Wakaki, Hiromi Mitsufuji, Yuki Computation and Language Artificial Intelligence Information Retrieval Multimedia Multimodal retrieval, which seeks to retrieve relevant content across modalities such as text or image, supports applications from AI search to contents production. Despite the success of separate-encoder approaches like CLIP align modality-specific embeddings with contrastive learning, recent multimodal large language models (MLLMs) enable a unified encoder that directly processes composed inputs. While flexible and advanced, we identify that unified encoders trained with conventional contrastive learning are prone to learn modality shortcut, leading to poor robustness under distribution shifts. We propose a modality composition awareness framework to mitigate this issue. Concretely, a preference loss enforces multimodal embeddings to outperform their unimodal counterparts, while a composition regularization objective aligns multimodal embeddings with prototypes composed from its unimodal parts. These objectives explicitly model structural relationships between the composed representation and its unimodal counterparts. Experiments on various benchmarks show gains in out-of-distribution retrieval, highlighting modality composition awareness as a effective principle for robust composed multimodal retrieval when utilizing MLLMs as the unified encoder. |
| title | MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval |
| topic | Computation and Language Artificial Intelligence Information Retrieval Multimedia |
| url | https://arxiv.org/abs/2510.15543 |