Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.17359 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909881516163072 |
|---|---|
| author | Li, Tianyuan Wang, Lei Ahmat, Ahtamjan Yang, Yating Ma, Bo Dong, Rui Han, Bangju |
| author_facet | Li, Tianyuan Wang, Lei Ahmat, Ahtamjan Yang, Yating Ma, Bo Dong, Rui Han, Bangju |
| contents | Generative cross-modal retrieval, which treats retrieval as a generation task, has emerged as a promising direction with the rise of Multimodal Large Language Models (MLLMs). In this setting, the model responds to a text query by generating an identifier corresponding to the target image. However, existing methods typically rely on manually crafted string IDs, clustering-based labels, or atomic identifiers requiring vocabulary expansion, all of which face challenges in semantic alignment or scalability.To address these limitations, we propose a vocabulary-efficient identifier generation framework that prompts MLLMs to generate Structured Semantic Identifiers from image-caption pairs. These identifiers are composed of concept-level tokens such as objects and actions, naturally aligning with the model's generation space without modifying the tokenizer. Additionally, we introduce a Rationale-Guided Supervision Strategy, prompting the model to produce a one-sentence explanation alongside each identifier serves as an auxiliary supervision signal that improves semantic grounding and reduces hallucinations during training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_17359 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MLLM-Driven Semantic Identifier Generation for Generative Cross-Modal Retrieval Li, Tianyuan Wang, Lei Ahmat, Ahtamjan Yang, Yating Ma, Bo Dong, Rui Han, Bangju Information Retrieval Generative cross-modal retrieval, which treats retrieval as a generation task, has emerged as a promising direction with the rise of Multimodal Large Language Models (MLLMs). In this setting, the model responds to a text query by generating an identifier corresponding to the target image. However, existing methods typically rely on manually crafted string IDs, clustering-based labels, or atomic identifiers requiring vocabulary expansion, all of which face challenges in semantic alignment or scalability.To address these limitations, we propose a vocabulary-efficient identifier generation framework that prompts MLLMs to generate Structured Semantic Identifiers from image-caption pairs. These identifiers are composed of concept-level tokens such as objects and actions, naturally aligning with the model's generation space without modifying the tokenizer. Additionally, we introduce a Rationale-Guided Supervision Strategy, prompting the model to produce a one-sentence explanation alongside each identifier serves as an auxiliary supervision signal that improves semantic grounding and reduces hallucinations during training. |
| title | MLLM-Driven Semantic Identifier Generation for Generative Cross-Modal Retrieval |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2509.17359 |