Unified Multimodal and Multilingual Retrieval via Multi-Task Learning with NLU Integration
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915742944854016 |
|---|---|
| author | Zhang, Xinyuan Zhang, Lina Chen, Lisung Liu, Guangyao Nie, Shuai Xu, Jiaming Shi, Runyu Huang, Ying Zhang, Guoquan |
| author_facet | Zhang, Xinyuan Zhang, Lina Chen, Lisung Liu, Guangyao Nie, Shuai Xu, Jiaming Shi, Runyu Huang, Ying Zhang, Guoquan |
| contents | Multimodal retrieval systems typically employ Vision Language Models (VLMs) that encode images and text independently into vectors within a shared embedding space. Despite incorporating text encoders, VLMs consistently underperform specialized text models on text-only retrieval tasks. Moreover, introducing additional text encoders increases storage, inference overhead, and exacerbates retrieval inefficiencies, especially in multilingual settings. To address these limitations, we propose a multi-task learning framework that unifies the feature representation across images, long and short texts, and intent-rich queries. To our knowledge, this is the first work to jointly optimize multilingual image retrieval, text retrieval, and natural language understanding (NLU) tasks within a single framework. Our approach integrates image and text retrieval with a shared text encoder that is enhanced by NLU features for intent understanding and retrieval accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_14714 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Unified Multimodal and Multilingual Retrieval via Multi-Task Learning with NLU Integration Zhang, Xinyuan Zhang, Lina Chen, Lisung Liu, Guangyao Nie, Shuai Xu, Jiaming Shi, Runyu Huang, Ying Zhang, Guoquan Information Retrieval Multimodal retrieval systems typically employ Vision Language Models (VLMs) that encode images and text independently into vectors within a shared embedding space. Despite incorporating text encoders, VLMs consistently underperform specialized text models on text-only retrieval tasks. Moreover, introducing additional text encoders increases storage, inference overhead, and exacerbates retrieval inefficiencies, especially in multilingual settings. To address these limitations, we propose a multi-task learning framework that unifies the feature representation across images, long and short texts, and intent-rich queries. To our knowledge, this is the first work to jointly optimize multilingual image retrieval, text retrieval, and natural language understanding (NLU) tasks within a single framework. Our approach integrates image and text retrieval with a shared text encoder that is enhanced by NLU features for intent understanding and retrieval accuracy. |
| title | Unified Multimodal and Multilingual Retrieval via Multi-Task Learning with NLU Integration |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2601.14714 |